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Abstract 


As  military  and  other  governmental  budgets  decline  and  impacted  project 
deadline  changes  require  instantaneous  responses,  cost  analysts'  tasks  become  more  and 
more  formidable.  Inaccurate  estimates  can  lead  to  misappropriation  of  resources  and  can 
thus  create  delays  in  goods  reaching  warfighters.  This  thesis  aims  to  avail  cost  estimators 
of  more  reliable  projection  tools  and  to  challenge  the  status  quo  of  cost  estimating,  the 
production  rate  cost  improvement  model,  when  programs  face  reductions  in  lot 
quantities.  The  findings  reveal  that  the  status  quo  proves  efficient  under  many  cost 
profiles,  but  clearly  does  not  estimate  as  well  when  a  program  suffers  lot  quantity 
reduction  coupled  with  loss  of  cost  efficiency.  Prior  research  recognized  the  importance 
of  changes  in  lot  quantity  to  cost  estimating,  but  definitive  guidance  never  surfaced  with 
regards  to  choosing  a  model.  Monte  Carlo  simulation  allows  us  to  vary  cost-affecting 
variables  and  isolate  conditions  where  the  use  of  a  fixed  cost,  cost  improvement  model 
provides  more  accurate  estimates  than  does  the  status  quo.  While  no  model  for 
estimation  should  be  discounted  without  exploration  of  its  usefulness,  we  argue  that  the 
fixed  cost  model  should  be  considered  for  use  based  on  its  ability  to  predict  increases  in 
average  unit  cost. 
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EVALUATION  OF  COST  IMPROVEMENT  MODELS  WHEN  PROGRAMS 
EXPERIENCE  UNPLANNED  PRODUCTION  DECREASES 


I.  Introduction 

Background 

Cost  analysts  face  formidable  obstacles  with  every  new  project  they  encounter. 
The  single  most  important  responsibility  of  any  cost  analyst's  job  is  to  make  sure  the 
calculations  and  figures  they  present  to  project  decision-makers  are  accurate  estimates  to 
the  best  of  their  abilities.  Regardless  of  exogenous  factors  weighing  on  cost  analysts, 
they  must  sift  through  all  information  and  develop  logical  conclusions  firmly  supported 
through  a  mixture  of  art  and  science.  As  military  and  other  governmental  budgets  decline 
and  project  deadline  changes  require  instantaneous  responses,  cost  analysts'  tasks  become 
more  and  more  difficult  and  are  subjected  to  intense  scrutiny  as  well  as  frequent 
criticism.  Such  a  climate  dictates  our  assignment:  to  evaluate  the  imperfections  in  the 
systems,  and  from  our  findings,  to  hone  reliable  and  effective  solutions  to  replace  the  less 
accurate  protocols  in  current  use.  Complacency  often  stands  in  the  way  of  providing  the 
best  possible  estimates;  relying  on  historical  processes  and  estimating  procedures  can 
limit  exposure  to  improvements  in  estimating  techniques.  We  must  challenge  the  status 
quo  whenever  feasible.  Fortunately,  technology  aids  us  in  our  exploration  of  new 
innovative  techniques.  Developing  expertise  with  estimating  software  now  fits  in  as  part 
of  our  job  description  and  is  absolutely  critical  to  accurate  estimates. 

Inaccurate  estimates  create  a  large  range  of  problems  for  program  managers  and 
other  decision-makers.  Current  and  future  budgets  base  resource  allocation  schemes  on 
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program  cost  estimates.  Should  the  estimates  be  inaccurate,  multiple  programs  face 
hardships;  resources  will  realistically  require  reallocation  and  the  overall  financial 
situation  will  be  constrained  impacting  current  projects  in  addition  to  future  endeavors. 
Furthermore,  as  decision-makers  face  funding  issues,  the  real  mission  failure  comes  when 
the  defense  acquisition  process  cannot  present  the  warfighter  with  products  they  expect 
and  need.  We  fear  that  Air  Force  cost  analysts  sometimes  use  incorrect  models  and 
create  bad  estimates,  which  generate  inefficiency  in  the  process.  Within  this  research 
work  we  are  attempting  to  further  perfect  the  overall  cost  analysis  process  and  current 
system  in  use  by  the  Air  Force.  Our  research  goal  is  to  provide  cost  analysts  and 
decision-makers  with  a  more  in-depth  analysis  of  cost  improvement  curve  theory  to  apply 
to  estimates. 

Purpose  of  This  Study 

Our  study  involves  a  very  specific  situation  we  cost  analysts  face.  In  the  current 
acquisition  environment,  budget  cuts  reduce  the  size  of  programs  and  the  new  cost 
estimates  must  adjust  for  the  new  fiscal  constraints.  Breaking  our  study  down  to  its 
simplest  form  yields  this  focus:  given  a  specific  cost  improvement  curve  estimation 
scenario  with  two  possible  models,  which  model  outperforms  the  other  when  predicting 
future  production  costs?  Research,  much  like  real  estimates,  never  turns  out  as 
uncomplicated  as  the  above  explanation,  but  that  simplistic  view,  of  necessity,  frames  our 
research. 

More  specifically,  our  research  looks  at  lot  production  quantities  that  decrease 
from  the  agreed  upon  quantity  after  initial  production  begins.  We  chose  this  situation 
because  as  budgets  become  more  constrained,  many  programs  lose  funding  and  reduce 
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order  quantities.  The  reverse  situation  of  increasing  order  quantities  could  theoretically 
happen  to  a  program,  but  is  unlikely  in  the  current  environment.  Given  this  scenario  we 
aim  to  find  the  best  possible  adjustments  to  the  original  estimate  in  order  to  capture  the 
effect  on  learning  and  production  rate  suffered.  “Conceptually,  production  rate  should  be 
expected  to  affect  unit  cost  because  of  the  impact  of  economies  of  scale”  (Moses, 

1990: 1).  These  costs  can  include  but  are  not  limited  to:  quantity  discounts  received  for 
ordering  larger  amounts  of  material;  reduced  ordering  and  processing  costs;  reduced 
shipping,  receiving,  and  inspection  costs  on  materials  ordered;  a  greater  use  of  facilities 
spreading  overhead  costs  over  output  quantity;  and  also,  the  inverse  of  these  costs 
(Moses,  1991:17-30).  These  effects  reach  beyond  the  range  of  a  simple  cost 
improvement  equation  and  call  for  the  addition  of  a  production  rate  variable.  We  refer  to 
these  equations  as  ‘production  rate’  and  ‘production  rate  adjustment’  models.  Dr.  David 
A.  Lee,  in  his  book  The  Cost  Analyst’s  Companion ,  presents  the  two  equations  we 
evaluate  in  our  research  (Lee,  1997:60-61). 


C(Q)  =  T1Qb(R/R0)c 

(1.1) 

C(Q)  =  (F/R)  +  TiQb 

(1.2) 

Dr.  Lee  offers  situations  where  each  equation  is  most  useful,  but  he  also  presents 
problems  with  each  of  the  equations  based  on  program  specifics.  Dr.  Lee  explains  that 
Equations  1.1  and  1.2  most  aptly  respond  to  the  cost  drivers  given  changes  in  production 
lot  sizes. 

Cost  analysts  must  estimate  their  respective  programs  with  both  science  and  art; 
we  expect  the  evaluation  of  these  two  equations  to  be  no  different.  Resources  available 
to  analysts,  including  the  Air  Force  Cost  Analysis  Handbook ,  do  not  outline  exactly  how 
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an  estimate  should  be  built.  The  resources  act  very  similar  to  a  ‘guide’  or  template  when 
developing  an  estimate.  We  intend  to  take  some  of  the  guesswork  out  of  developing  a 
cost  estimate  with  this  research  by  revealing  which  equation  performs  better  with 
decreasing  production  quantities.  While  predicting  future  events  cannot  be  100% 
accurate,  we  can  provide  indicators  showing  analysts  a  better  estimating  framework. 
Research  Questions 

Given  the  above  scenario,  our  two  equations,  and  the  analysis,  we  answer  the 
following  research  questions: 

1.  What  is  the  current  practiced  method  for  rate  adjustments  when  lot  order 
quantities  are  changed  from  the  manufacturing  plant’s  designed  buy  quantity  and 
is  that  method  consistent  across  the  field? 

2.  What  factors/inputs  influence  each  of  the  specified  cost  improvement  curve 
equations  when  lot  quantities  change? 

3.  Which  of  these  cost  improvement  models  best  estimates  the  impacts  of  changes  in 
production  buy  quantities  and  what  are  the  driving  forces  behind  each  of  the 
estimates? 

General  Approach 

In  order  to  answer  the  research  questions  mentioned  above,  we  conducted  an 
extensive  literature  review  to  uncover  the  progression  of  cost  improvement  curves, 
production  rate  adjustments,  and  the  analysis  of  these  models.  Based  on  the  research,  we 
performed  a  Monte  Carlo  analysis  to  evaluate  each  of  the  equations  and  to  conclude 
which  inputs  to  the  production  process  influence  the  estimate.  We  also  use  historic 
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program  data  to  verify  our  findings  and  to  better  align  our  research  to  actual  cost 
estimates.  Our  cumulative  findings  allow  us  to  draw  conclusions  about  the  application  of 
each  of  the  equations  to  a  cost  estimate,  and  hopefully  to  aide  estimators  in  making  more 
accurate  estimates. 

For  the  purpose  of  our  study,  we  will  be  using  the  term  ‘cost  improvement  curve’ 
to  describe  the  phenomenon  often  referred  to  as  learning  curve,  production  cost  progress 
curve,  cost-quantity  curves,  experience  curves  and  cost-progress  curves  (Department, 
2007:8-1).  We  chose  this  terminology  because  the  term  ‘cost  improvement’  encompasses 
factors  beyond  the  standard  concept  of  ‘learning.’  While  we  understand  the  basic 
structure  of  cost  improvement  curves,  there  have  been  alterations  to  the  original  methods 
to  account  for  many  possible  production  situations.  Academics  found  the  original  cost 
improvement  equations  needed  elaboration  because  many  other  factors  can  affect  the 
amount  of  learning  that  takes  place  in  a  production.  Though  in  general  practice  the 
cumulative  quantity  is  the  main  cost  driver,  and  as  quantity  increases  the  unit  cost 
decreases,  other  cost  drivers  are  present.  In  the  case  of  production  rate,  there  is  an 
inverse  relationship  between  production  rate  and  unit  cost.  As  production  rate  increases, 
the  plant  should  gain  economies  of  scale  and  decrease  unit  cost  (and  the  reverse  situation 
should  also  hold  true). 

Assumptions 

1.  We  assume  at  the  point  of  changing  future  lot  sizes  the  product  requirements  remain 
constant  throughout  the  rest  of  the  programs  life.  Our  simulated  data  and  our  equation 
prediction  error  will  be  built  upon  the  assumption  of  stable  product  requirements. 
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Requirements  may  change  during  production  of  an  item,  which  would  call  for  a  new  cost 
estimate  based  on  the  changing  information.  We  do  not  model  that  scenario  because 
providing  an  encompassing  range  of  possible  requirement  changes  is  not  feasible. 

2.  Based  on  our  research,  the  two  equations  that  Dr.  Lee  presents  best  estimate  cost 
improvement  when  a  program  suffers  changes  in  production  lot  sizes.  Other  research 
also  led  us  to  this  conclusion,  but  there  may  be  a  need  to  explore  new  equations  never 
presented.  We  will  not  be  measuring  prediction  accuracy  of  every  cost  improvement 
equation  even  though  there  may  be  specific  conditions  with  our  simulated  and  actual  data 
where  other  models  outperform  our  models.  We  did  identify  the  need  to  model  two  other 
cost  improvement  models  found  in  the  literature  review,  but  the  addition  of  these  models 
does  not  exhaustively  collect  all  possible  cost  improvement  models  and  their  predictive 
capabilities.  Across  the  spectrum  of  varying  conditions  within  our  scenario  we  assume 
our  models  will  consistently  outperform  other  models. 

Limitations 

1.  Our  simulated  data  and  actual  data  cannot  cover  every  possible  situation  analysts  may 
face.  We  generalize  our  findings  to  real  situations,  but  we  cannot  be  certain  that  other 
factors  outside  the  range  of  our  study  are  driving  cost.  Our  simulated  data  represents  a 
‘normalized’  data  set  where  abnormalities  have  been  removed. 

2.  Along  the  same  lines  as  our  first  assumption,  we  cannot  possibly  research  all 
exogenous  factors  affecting  cost.  We  limit  ourselves  to  the  most  common  cost  drivers, 
but  we  understand  other  actual  programs  can  face  unique  situations.  The  burden  falls  on 
the  program  analyst  to  filter  the  program  and  decide  what  factors  are  truly  driving  cost, 
and  if  our  research  can  be  useful. 


6 


Chapter  Summary 

Chapter  I  provided  an  overview  of  our  research  and  states  the  research  questions 
we  hope  to  answer  through  our  results.  In  Chapter  II  we  review  the  history  of  cost 
improvement  curves  and  the  advancements  researchers  have  made  to  understand  and 
evaluate  the  effects  of  production  rates.  Based  upon  the  information  provided  in  the 
literature  review,  we  outline  in  Chapter  III  our  methods  for  creating  a  robust  Monte  Carl 
simulation  and  evaluation  of  Equations  1.1  and  1.2.  Chapter  IV  will  show  the  results  and 
analysis  of  our  Monte  Carlo  simulation  and  evaluation.  Finally,  Chapter  V  will 
summarize  the  significant  findings  of  our  analysis  and  highlight  potential  policy 
implications  to  consider. 
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II.  Literature  Review 


We  target  two  areas  in  the  literature  review:  first,  an  overview  of  cost 
improvement  curves  to  include  production  rate  variations,  and  second,  previous  research 
performed  on  production  rate  variations.  Academics  have  produced  extensive  research 
on  learning  curve  theory  and  application,  but  we  will  only  focus  on  research  that  directly 
relates  to  the  production  rate  variations.  Our  literature  review  does  not  serve  as  a  stand¬ 
alone  document  on  all  variations  to  learning  curve  models;  for  a  more  in  depth 
explanation  of  learning  curves,  readers  should  reference  the  many  available  publications. 
An  Overview  of  Cost  Improvement  Curve  Theory 

As  previously  mentioned,  for  the  sake  of  continuity  we  will  be  using  the  term 
‘cost  improvement  curve’  instead  of  any  of  the  other  acceptable  variations.  Though  we 
often  use  the  names  interchangeably,  there  are  subtle  differences  in  the  phrasing  and 
meaning  of  each  variation.  For  example  ‘learning’  describes  the  efficiencies  gained  by 
laborers  improving  performance  at  producing  an  item  in  a  repetitive  process,  while 
‘production  cost  progress’  explains  the  process  of  repetitive  production  where  an  increase 
in  the  total  quantity  produced  may  lower  the  unit  cost  of  each  item.  Any  recurring  (or 
variable)  production  costs,  including  labor,  raw  materials,  and  manufacturing  costs  of  an 
individual  item  decrease  as  the  total  quantity  of  items  produced  increases  (Department, 
2007:8-5).  The  original  forms  of  the  learning  curve  model  do  not  include  fixed  costs  in 
the  equation,  but  as  we  will  discuss  later,  fixed  costs  can  contribute  to  model  accuracy 
under  certain  circumstances.  ‘Improvement’  refers  to  the  over- arching  efficiencies  that 
sometimes  cannot  be  pinpointed  but  occur  in  a  repetitive  process  (Department,  2007:8-2). 
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Each  variation  of  the  cost  improvement  curve  involves  an  important  aspect  of  production 
and  our  scenario  encompasses  many  of  these  variables. 

Cost  improvement  models  serve  as  crucial  tools  for  cost  analysts  across  all  fields. 
The  aircraft  industry  first  recognized  the  usefulness  of  cost  improvement  models, 
uncovering  and  utilizing  the  predictive  value  of  modeling  learning  curves.  Cost  analysts 
discovered  that  these  models  could  be  applied  almost  universally  across  production  and 
thus,  other  industries  soon  followed  in  practice  (Department,  2007:8-4).  Generalization 
of  the  learning  concepts  allows  for  use  of  the  models  in  calculating  expected  labor  hours, 
resources,  and  costs.  “In  manufacturing,  learning  curve  representations  are  used  to  plan 
manpower  needs,  set  labor  standards,  establish  sales  prices,  aid  make/buy  decisions, 
judge  wage  incentive  payments,  evaluate  organizational  efficiency,  develop  quantity 
sales  discounts,  analyze  employee  training  programs,  evaluate  capital  equipment 
proposals,  predict  future  production  unit  costs,  and  create  production  delivery  schedules” 
(Smith,  1989:1).  The  application  possibilities  range  across  many  different  decision 
points  within  the  acquisition  process.  The  Air  Force  utilizes  cost  improvement  curve 
estimates  across  a  range  of  systems  and  acquisitions  to  include  airframes,  modifications, 
common  avionics  acquired  for  multiple  platforms,  engines,  missiles,  and  satellite 
hardware  (Department,  2007:8-2).  Accurate  estimation  plays  an  important  role  for  cost 
analysts  and  provides  valuable  perspective  as  they  evaluate  contractor  proposals  as  well. 
The  effects  of  misestimating  can  negatively  affect  the  decision-making  process;  thus,  the 
estimates  must  be  as  accurate  as  possible. 
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The  two  most  common  versions  of  the  cost  improvement  curve  models  are  the 
cumulative  average  model  developed  by  T.P.  Wright,  and  the  unit  cost  theory  formulated 
by  James  R.  Crawford  and  based  on  Wright’s. 

The  Cumulative  Average  Model 

In  1936,  aircraft  researcher  T.P.  Wright  became  the  father  of  learning  curve 
theory  when  he  created  a  model  explaining  learning-related  reductions  in  airframe 
construction  costs.  The  basic  premise  effectively  illustrated  that  when  the  number  of 
aircraft  produced  in  sequence  doubled,  the  cumulative  average  direct  labor  input  per 
aircraft  decreased  in  a  regular  pattern.  His  ratio  relationship  could  be  modeled 
exponentially,  but  also  became  a  linear  function  when  applied  to  the  labor/cost  changes 
that  occurred  over  the  sequence  of  production  units  (Department,  2007:8-4).  The  linear 
function  represented  Wright’s  ‘learning  curve  slope’. 

Wright’s  theory  can  be  expressed  mathematically  as: 

A(Q)  =  A1(2c)b  (2.1) 

where 

A(Q)  =  average  cost  to  produce  the  first  Q  units 
Ai  =  first  unit  cost  (model  parameter) 

Qc  =  total  quantity  of  units  produced  (whose  average  cost  is  to  be  computed) 
b  =  slope  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

Equation  2.1’s  framework  does  not  appear  complex,  but  conceptually  there  are 
two  essential  elements  that  must  be  recognized  and  accepted.  First,  the  value  of  Ai  is 
described  as  the  ‘first  unit  cost’  but  is  not  the  actual  cost  of  the  first  unit  of  production. 

Ai  is  an  estimated  model  parameter  derived  from  historical  values  to  fit  the  curve. 
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Second,  the  slope  refers  to  the  rate  of  learning,  and  represents  the  percentage  by  which 
reoccurring  labor/cost  decreased  every  time  production  quantity  doubles.  “For  example, 
for  a  slope  of  80%,  the  value  of  labor/cost  for  every  doubling  of  the  quantity  Q  is  80%  of 
the  value  for  Q;  equivalently,  every  time  the  production  quantity  doubles,  the  associated 
hours/cost  improves  at  (is  reduced  by)  a  rate  of  20%”  (Department,  2007:8-6). 

Figure  2.1  below  depicts  the  above  scenario  of  80%  slope,  starting  with  a  unit  1 
cumulative  cost  of  $1,000. 


Cum  Avg 


Figure  2.1:  Wright  Model  Plotted  on  Arithmetic  Grids 


The  Unit  Cost  Model 

Following  WWII,  James  R.  Crawford  updated  Wright’s  model  based  on 
information  from  aircraft  production  during  the  war.  Very  similarly  to  Wright,  Crawford 
theorized  that  the  cost  per  unit  decreases  by  some  constant  percentage  (ratio)  as  the  total 
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number  of  units  produced  doubles.  Crawford  found  this  model  more  useful  for  his 
applications. 

The  mathematical  expression  of  Crawford’s  model  is  identical  to  Wright’s  model 
though  Crawford’s  definition  of  terms  differs. 

C(Q)  =  T!eb  (2.2) 

where 

C(Q)  =  cost  to  produce  the  Qth  unit 
Ti  =  first  unit  cost  (model  parameter) 

Q  =  unit  number  (whose  cost  is  to  be  computed) 
b  =  slope  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

The  subtle  difference  from  Equation  2.1  is  notable:  in  Equation  2.2  the  dependent 
variable  C(Q)  is  the  cost  of  a  specific  unit,  while  in  Equation  2.1  the  cost  is  an  average 
across  all  prior  units.  These  distinctions  are  reflected  in  the  title  description  of  each 
model.  Figure  2.2  below  plots  the  unit  cost  curve  with  a  learning  slope  of  80%  and  a  unit 
1  cost  of  $1,000.  Figure  2.1  above  looks  identical  to  Figure  2.2;  the  only  differences  are 
the  axes  labels  and  the  interpretations  of  the  data. 
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Unit 

Avg  Cost  Per  Unit 


Figure  2.2:  Crawford  Model  Plotted  on  Arithmetic  Grids 
Unit  Versus  Cumulative  Average 

Whether  one  model  outperforms  the  other  model  depends  on  the  situation,  and 
through  mathematical  manipulation  each  model  can  produce  the  other’s  results.  While 
the  equations  can  be  applied  interchangeably  for  estimates,  the  form  we  choose  must 
remain  constant  throughout  the  estimate  for  accuracy  (Anderson,  2003).  For  example,  if 
the  analyst  begins  the  estimate  with  the  unit  cost  model,  he/she  must  complete  all 
calculations  using  that  chosen  model  and  define  the  results  in  unit  cost  form.  If  he/she 
wishes  to  explain  cumulative  average  results,  that  application  must  be  employed  from  the 
beginning,  or  else  the  final  numbers  must  be  changed  through  addition  of  each  unit  cost 
(Anderson,  2003).  Analysts  must  remain  true  to  the  use  of  their  chosen  cost 
improvement  curves  in  order  to  create  valid  estimates.  These  two  models  remain  the 
most  commonly  applied  models  because  of  their  simplicity  and  consistent  performance. 
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When  the  two  models  are  compared  side  by  side  they  give  the  results  seen  in 


Table  2.1  and  Figure  2.3. 

Table  2.1.  Comparison  of  Cumulative  Average  and  Unit  Formulation  (Shea,  1994:18) 


Cum  Avg  Cost  Formulation 
(Constant  Year  $'s) 

80%  Slope 

Unit  Cost  Formulation 
(Constant  Year  $'s) 

80%  Slope 

Unit  # 

Cum  Avg  Cost 

Unit  Cost 

Unit  # 

Cum  Avg  Cost 

Unit  Cost 

1000 

1000 

1000 

1000 

800 

600 

900 

800 

702 

506 

834 

702 

640 

454 

786 

640 

596 

418 

748 

596 

562 

392 

717 

562 

7 

534 

371 

7 

691 

534 

8 

512 

355 

8 

668 

512 

493 

341 

649 

493 

477 

329 

632 

477 

Unit  Cost  Theory  Cost  Per  Unit  —  ft-  Cum  Avg  Theory  Cost  Per  Unit 


Figure  2.3:  Cum  Avg.  Theory  and  Unit  Cost  Theory  Plotted  By  Incremental  Unit  Cost 

(Shea,  1994:20) 
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When  looking  at  Figure  2.3,  “the  cumulative  average  cost  declines  by  a  constant 
percentage  between  double  quantities;  however,  when  converted  to  incremental  unit 
costs,  the  percent  decline  is  non-constant”  (Shea,  1994:18).  The  converse  holds  true  for 
the  unit  cost  theory,  where  the  unit  cost  declines  at  a  constant  percentage  and  the 
cumulative  average  cost  does  not  decline  at  a  constant  percentage.  In  reality,  creating  the 
two  models  will  show  variance  because  the  unit  1  cost  will  not  be  the  same,  but  the 
information  highlights  the  differences  of  the  two  equations.  Again,  the  important  fact 
remains  that  cost  analysts  must  remain  consistent  in  the  application  of  whichever 
equation  form  they  choose  to  employ. 

Theoretically,  both  the  cumulative  average  model  and  the  unit  cost  model  have 
validity.  The  basic  premise  that  a  laborer’s  repetition  of  task  over  and  over  again  will 
cause  the  laborer  to  become  better  at  the  task  can  be  perceived  in  performance.  For 
example,  an  observer  could  theoretically  stand  next  to  the  laborer  with  a  stopwatch  and 
could  measure  the  changes  in  time  taken  to  complete  a  repetitive  process.  The  efficiency 
of  motion  and  task  completion  gained  works  both  in  theory  and  in  practice.  As  we  look 
at  variations  to  the  two  original  models  and  eventually  production  rate  adjustment 
models,  the  cumulative  average  model  and  the  unit  cost  model  still  demonstrate  validity, 
even  though  there  is  no  feasible  way  to  physically  measure  all  of  the  inputs  and  outputs. 
Linear  Transformation 

Both  the  cumulative  average  model  and  the  unit  cost  model  are  commonly 
expressed  in  log-linear  form  for  statistical  evaluation  purposes.  When  the  exponential 
curves  are  transformed  to  their  linear  forms,  they  become  more  easily  understood  in 
presentation  and  practice.  The  log-linear  form  is  expressed  as: 
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Log  A(Q)  =  log  Ai  +  b  log  Q 


(2.3) 


When  graphically  represented,  this  equation  illustrates  the  linear  relationship  between 
cost  and  learning  that  the  original  forms  of  the  equation  embody.  If  we  take  a  unit  cost  of 
$1,000  and  a  learning  slope  of  80%  just  as  before,  and  plot  the  data  on  a  logarithmic 
scale,  the  results  are  shown  in  Figure  2.4  below. 


Figure  2.4:  Unit  Cost  Curve  Plotted  on  Logarithmic  Grids 

Figure  2.4  not  only  gives  the  same  information  as  the  arithmetic  grid  plots,  but 
also  can  be  used  to  better  illustrate  the  relationship  between  learning  slope  and  cost 
necessary  for  analysis  of  the  equations  statistically. 
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Variations  to  Original  Models 

The  wide  range  of  applications  for  cost  improvement  curve  theory  opens  the  door 
to  variations  that  more  aptly  model  specific  situations.  Two  equations  are  no  longer 
sufficient  to  handle  the  nature  of  a  product,  manufacturing  process,  business 
environment,  or  countless  other  factors  that  reach  beyond  typical  learning.  Because  we 
do  not  have  clear  guidance  on  which  equations  to  use,  we  must  filter  through  all 
possibilities  and  make  determinations  based  on  program  specifics  and  indicators.  Dr. 
Adedeji  B.  Badiru  published  an  extensive  compilation  of  ‘univariate’  and  ‘multivariate’ 
learning  curve  models.  ‘Univariate’  equations,  exemplified  by  Equations  2.1  and  2.2, 
calculate  projections  using  single  input  variables;  ‘multivariate’  equations  utilize  more 
than  one  input  variable  such  as  processes  using  Equations  1.1  and  1.2  in  Chapter  I. 

While  these  models  are  not  the  focus  of  our  research,  the  principles  driving  their 
conception  and  application  aid  the  understanding  of  learning  curve  theory  and  how  such 
theory  is  applied.  Simply,  Dr.  Badiru  concludes  that  there  are  underlying  causes,  often 
unseen  and  not  directly  traceable,  that  affect  cost;  these  models  aim  to  capture  the  cost 
changes  when  the  programs  exhibit  certain  symptoms. 

The  classical  developments  based  on  the  Wright  and  Crawford’s  original  models 
include: 

-  The  S -Curve 

-  The  Stanford-B  Model 

-  DeJong’s  Learning  Formula 

-  Levy’s  adaptation  formula 

-  Glover’s  learning  formula 

-  Pagel’s  exponential  function 

-  Knecht’s  upturn  model 

-  Yelle’s  product  model 

-  Multiplicative  Power  Model 
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Each  of  these  adaptations  to  the  model  may  outperform  the  original  models  given  certain 
program  specifics.  The  cost  analysts  must  investigate  the  specifics  of  their  respective 
programs  and  accordingly  apply  the  appropriate  learning  curve  equations.  In  the 
decision-making  process,  these  equations  can  improve  analysts’  evaluations  of  the 
designs  of  training  programs,  the  manufacturing  economic  analyses,  the  breakeven 
analyses,  the  make  or  buy  decisions,  the  manpower  scheduling,  the  production  planning, 
the  labor  estimating,  the  budgeting,  and  the  resource  allocation  (Badiru,  1991:439-440). 
Dr.  Badiru  also  explains  how  analysts  must  choose  between  a  ‘univariate’  (a  single  input 
usually  quantity)  and  a  ‘multivariate’  model  for  a  cost  estimate.  The  choice  rests  on 
many  factors  related  to  the  actual  calculations  as  well  as  to  the  delivery  of  the 
information  in  an  intelligible  form  to  decision-makers.  Based  on  the  amount  of  data  and 
the  time  and  statistical  software  available,  the  use  of  multivariate  models  might  not  be 
possible.  Univariate  models  can  be  applied  competently  with  limited  data,  but  these 
models  may  not  be  capturing  all  variables  affecting  cost.  Multivariate  models  require  a 
better  knowledge  base  of  statistical  data  for  starters,  as  well  as  expertise  in  the  art  form  of 
presenting  the  information.  Sometimes  a  parsimonious  solution  proves  to  be  more  useful 
in  the  decision-making  process  as  many  non-analysts  may  prove  unable  to  accurately 
comprehend  complex  models.  Since  both  of  the  models  we  evaluate  in  this  research  are 
multivariate  production  rate  models,  science  and  art  alike  must  be  utilized. 


18 


Cost  Improvement  Curve  Production  Rate 

Expansion  of  the  original  cost  improvement  curve  equations  called  for  a  term  to 
explain  and  model  effects  when  manufacturing  production  rates  change  throughout  the 
production  lifecycle.  Hoffmayer’s  1974  hypothesis,  in  a  report  completed  at  RAND, 
proposed  that  as  production  rate  increased,  manufacturers  should  gain  greater  efficiency 
above  only  ‘learning’,  and  that  the  resulting  effect  would  be  a  decrease  in  unit  cost 
(Hoffmayer,  1974:2).  Decreases  in  unit  cost  can  be  contributed  to  “greater  specialization 
of  labor,  quantity  discounts  and  efficiencies  associated  with  raw  materials  purchases,  and 
greater  use  of  facilities  permitting  fixed  overhead  costs  to  be  spread  over  a  larger  output” 
(Moses,  1991:2).  At  the  time  of  the  Hoffmayer  study,  cost  analysts  did  not  fully 
understand  the  concept  of  production  rate  and  how  it  affected  cost  improvement.  More 
recently,  the  Air  Force  Cost  Analysis  Handbook  recognizes  production  rate’s  link  to 
economies  and  diseconomies  of  scale  as  the  production  rate  increases  and  decreases 
(Department,  2007:8-31). 

Hoffmayer’s  study  recognized  the  importance  of  production  rate  and  the 
widespread  utility  an  accurate  tool  to  predict  cost  could  provide  cost  analysts.  Karl 
Hoffmayer  and  the  other  RAND  authors  aimed  to  create  an  estimating  model  to  capture 
the  magnitude  of  costs  and/or  cost  savings  realized  through  changes  in  production  rate 
(Hoffmayer,  1974:1).  To  accomplish  their  goal,  the  RAND  study  authors  focused  on 
how  production  rate  would  affect  four  major  cost  elements:  manufacturing  labor, 
materials,  tooling,  and  engineering  (Hoffmayer,  1974:1).  Along  with  major  cost 
elements,  the  RAND  study  focused  on  manufacturing  overhead,  which  had  previously 
been  omitted  from  cost  improvement  (learning  curve)  equations.  Beginning  with  these 
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five  areas,  the  authors  hypothesized  that  they  could  find  other  exogenous  factors,  outside 
of  the  program’s  control,  that  altered  unit  cost  with  production  rate. 

Hoffmayer’s  study  determined  the  causes  of  changes  in  production  rate  to  be 
design  problems,  cost  growth,  funding  problems,  modifications,  and  other  similar  factors. 
These  same  causes  are  prevalent  throughout  the  current  acquisition  process  and  have 
directly  led  to  the  scenario  we  are  studying,  as  well  as  to  the  two  production  rate 
adjustment  equations  we  are  evaluating.  The  unresolved  cost-related  problems  within 
acquisition  programs  include:  the  way  in  which  rate  changes  were  and  are  achieved;  the 
availability  of  suppliers;  the  local  labor  supply;  management  policy;  the  timing  of  rate 
changes;  plant  capacity;  plant  backlog;  and  a  number  of  other  transitory  factors 
(Hoffmayer,  1974:41).  These  elements  drastically  changed  the  major  cost  components 
and  overhead  allocation.  Hoffmayer  concluded  that  while  he  and  the  other  authors  could 
understand  the  concepts  involved  in  production  rate  changes,  they  were  unable  to  create  a 
useful  model  fit  for  all  scenarios  (Hoffmayer,  1974:41).  The  most  important  finding  of 
Hoffmayer’s  study  related  to  the  relationship  between  production  rate  changes  and 
overhead. 

Hoffmayer  and  the  authors  of  the  RAND  study  looked  at  major  acquisition 
programs  and  concluded  the  one  cost  element  that  is  clearly  a  function  of  production  rate 
is  overhead.  The  effect  presented  itself  clearly  through  the  RAND  study;  because 
overheard  costs  could  reach  upwards  of  50%  of  total  cost,  even  minute  changes  could 
appear  significant.  The  RAND  study  finally  concluded  that,  “When  the  total  volume  of 
business  is  very  low,  cost  can  be  quite  sensitive  to  [production]  rate.  When  total  volume 
is  high,  the  influence  of  rate  is  reduced  but  still  perceptible”  (Hoffmayer,  1974:43).  Even 


20 


following  conclusion  of  their  study,  the  authors  found  themselves  unable  to  predict  the 
production  rate  changes  with  confidence,  and  thus  they  could  not  fully  estimate  cost  with 
any  degree  of  certainty.  The  RAND  study  did  frame  the  idea  of  production  rate  changes 
and  led  to  further  investigation  and  statistical  analysis  to  create  a  useable  estimating 
model.  The  Equations  1.1  and  1.2  we  are  evaluating  in  our  study  are  based  on  the  same 
concept  found  in  the  RAND  study. 

Production  Rate  Model 

Production  rate  models  evolved  based  on  the  principles  described  above.  Many 
cost  estimating  handbooks  now  include  sections  describing  production  rates  and  the 
effects  they  will  bear  on  cost  estimates.  The  Air  Force  Cost  Analysis  Handbook  presents 
the  most  common  cost  improvement  model  with  a  production  rate  term  (Department, 
2007:8-31).  Equation  2.4  shows  the  production  rate  model. 

C(Q)  =  TiQb  Rc  (2.4) 

where 

C(Q)  =  cost  to  produce  the  Qth  unit 
Tj  =  first  unit  cost  (model  parameter) 

Q  =  unit  number  (whose  cost  is  to  be  computed) 
b  =  slope  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

R  =  production  rate  (number  of  units  produced  in  a  production  period) 
c  =  rate  coefficient  (model  parameter)  =  ln(slope)/ln(2) 
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Given  production  rate  changes,  Equation  2.4  outperforms  the  basic  learning  curve 
model  (Moses,  1990:30).  The  problem  rests  in  determining  which  independent  variables 
drive  the  effectiveness  of  the  equation  and  subsequently  allow  analysts  to  predict  the 
equation’s  usefulness.  “Conceptually,  production  rate  should  be  expected  to  affect  unit 
cost  because  of  the  impact  of  economies  of  scale.  Higher  production  rates  may  lead  to 
several  related  effects:  greater  specialization  of  labor,  quantity  discounts  and  efficiencies 
associated  with  raw  materials  purchases,  and  greater  use  of  facilities  permitting  fixed 
overhead  costs  to  be  spread  over  a  larger  output  quantity”  (Moses,  1990:1-2).  Adding  a 
production  rate  term  to  the  cost  improvement  model  also  creates  clear  disadvantages. 
Finding  a  clear  production  rate/rate  slope  to  use  in  the  model  is  the  first  disadvantage  of 
the  production  rate  model.  This  problem  can  be  minimized  and  possibly  resolved 
through  statistical  analysis.  A  second  disadvantage  occurs  if  the  production  rate  is 
constantly  increasing  or  decreasing.  This  situation  causes  high  co-linearity  between  the 
unit  and  rate  variables.  No  clear  solution  for  this  problem  can  be  found,  so  analysts  must 
work  the  individual  problems  according  to  the  specifics  of  their  data  set  and  program. 

The  limit  on  reductions  or  gains  in  production  is  viewed  as  another  disadvantage  to  the 
production  rate  model.  Once  a  plant  reaches  either  minimum  or  maximum  capacity, 
large  expenditures  will  take  place:  overtime,  expedited  material  orders,  purchase  of  new 
capital,  hiring  more  of  the  labor  force,  and  increased  training  (Lee,  1997:60).  Due  to 
these  inadequacies,  some  cost  analysts  may  find  themselves  hesitant  to  utilize  the 
production  rate  model. 
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Evaluation  of  the  Production  Rate  Model 


In  a  report  completed  at  the  Naval  Postgraduate  School  in  1990,  O.  Douglas 
Moses  evaluated  the  Production  Rate  Model.  The  report.  Learning  Curx’e  and  Rate 
Adjustment  Models:  Comparative  Prediction  Accuracy  Under  Varying  Conditions , 
compared  the  original  unit  cost  Model  (Equation  2.2)  developed  by  Crawford  and  the 
production  rate  model  (Equation  2.4)  to  find  out,  simply,  which  equation  performs  better. 
Based  on  available  research,  Moses  hypothesized  that  a  clear  relationship  existed 
between  cost  and  production  rate,  but  since  the  relationship  would  vary,  neither  equation 
outperformed  the  other  outright  (Moses,  1990:3).  Unable  to  rely  on  prior  research  and 
practical  application  because  he  could  not  verify  the  proper  application  of  the  production 
rates  used,  Moses  created  simulated  data  to  evaluate  the  equations. 


Table  2.2.  Moses’  (1990)  Independent  Variable  Values  For  Data  Simulation  (Moses,  1990:13) 


Variable 

Levels/V  alues 

Data  History 

4 

7 

10 

Variable  Cost  Learning  Rate 

75% 

85% 

95% 

Fixed  Cost  Burden 

15% 

33% 

50% 

Production  Rate  Trend 

Level 

Growth 

Production  Rate 

Instability/V  ariance 

.05 

.15 

.25 

Cost  Noise/Variance 

.05 

.15 

.25 

Future  Production  Level 

Low 

Same 

High 
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Moses  derived  a  generic  cost  function  to  create  his  simulated  data,  but  only  solved 
the  cost  function  for  a  set  number  of  discrete  values  as  seen  in  Table  2.2  above.  True  cost 
functions  do  not  exist  in  the  manufacturing  process.  If  manufacturers  knew  their  exact 
cost  function,  then  analysts  would  not  need  to  develop  their  own  equations.  As  a  result, 
Moses  derived  his  equation  from  cost  components.  Moses’  equation  allowed  him  to 
inject  various  independent  variables  into  the  equation  at  different  levels  to  create 
simulated  cost  data.  Using  the  simulated  cost  data,  Moses  assessed  each  model 
individually  and  measured  the  error  in  predicting  the  future  lot  costs  (also  created  with 
the  cost  function).  The  strength  of  Moses’  generic  cost  function  and  independent 
variables  validates  the  research  and  his  findings.  We  label  his  cost  simulation  function 
Equation  2.5  and  provide  the  definition  of  terms  below. 

C(Q)  =  VCi(Qd)  +  SFC  (PR'1)  (2.5) 

where 

C(Q)  =  unit  cost 

VC  =  variable  cost  per  unit  (learning  included) 

SFC  =  standard  fixed  cost  per  unit 
Q  =  cumulative  quantity 

d  =  parameter,  the  learning  index  (same  as  learning  slope) 

PR  =  production  rate  for  any  period 

The  above  cost  function  encompasses  the  independent  variables  Moses  uses  to 
explore  the  cost  drivers  of  each  equation.  His  independent  variables  included  data 
history,  variable  cost  learning  rate,  fixed  cost  burden,  production  rate  trend,  production 
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rate  instability/variance,  cost  noise/variance,  and  future  production  level  (Moses, 

1990: 13).  Moses  chose  these  variables  because  they  most  resemble  variables  analysts 
could  flag  in  their  programs  as  indicators  identifying  which  cost  improvement  curve  to 
apply. 

Moses  did  not  find  overwhelming  evidence  demonstrating  that  either  tested 
equation  outperforms  the  other  across  the  entire  range  of  scenarios.  Rather,  Moses  found 
that  under  certain  conditions  each  equation  estimates  production  rate  superiorly  to  the 
other.  Moses  discovered  that  the  following  general  tendencies  led  to  reduction  in  the 
incidence  of  prediction  errors  and  to  improved  accuracy  for  the  production  rate  equation: 

-  The  number  of  observations  available  for  the  analysis  was  relatively  high 

-  The  amount  of  fixed  cost  in  total  cost  was  relatively  high 

-  The  production  rate  trend  had  been  growing  during  the  model  estimation  period 

-  The  period-to-period  variability  in  production  rate  was  relatively  large 

-  Random  noise  in  cost  due  to  unsystematic  factors  impacting  cost  was  relatively  low 

-  Production  volume  was  expected  to  be  cutback  in  the  future  periods  for  which  cost 
predictions  were  being  made  (Moses,  1990:29). 

Moses  also  recognized,  “The  greatest  impact  (of  changes  in  the  various  factors) 

on  relative  prediction  accuracy  (of  the  learning  curve  approach  and  the  rate  adjustment 

approach)  occurs  when  cutbacks  in  future  production  are  anticipated”  (Moses,  1990:30). 

Based  on  this  finding,  the  most  crucial  model  selection  decisions  present  themselves  in  a 

program  where  future  production  volume  is  declining  (Moses,  1990:30).  Our  research 

deals  directly  with  the  above  situation.  We  must  acknowledge  that  Moses  was  aware  of 

the  impact  decreases  in  production  rates  can  have  on  cost  estimates. 

By  astutely  analyzing  the  interactions  of  independent  variables,  Moses’  research 

also  discovered  that  the  factors  on  the  relative  prediction  tend  to  be  additive.  The 

analysis  did  not  reveal  significant  scenarios  where  the  basic  learning  curve  function 
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outperformed  the  production  rate  model.  The  Crawford  model  did  outperform  the 
production  rate  model,  but  not  consistently  nor  convincingly.  Because  Moses  created 
simulated  data  at  discrete  values  and  tested  the  models  against  that  data,  there  is  no  way 
to  know  if  his  findings  can  be  generalized  outside  of  that  specific  dataset.  Our  research 
fills  this  void  by  using  both  Monte  Carlo  simulation  to  create  an  inclusive  range  of 
possible  scenarios  and  historical  program  data  to  evaluate  the  production  rate  adjustment 
equations. 

Including  Fixed  Costs  in  Cost  Improvement  Curves 

The  longstanding  mindset  of  ‘learning  curve’  models  states  that  the  models  can  be 
applied  only  to  recurring  costs.  As  learning  curves  evolve  into  to  cost  improvement 
curves,  this  rule  reflects  that  change.  Based  on  the  above  research  done  by  Hoffmayer 
and  Moses,  large  fixed  costs  factored  as  a  proportion  of  total  cost  can  drastically  skew  the 
results  of  the  original  cost  improvement  models  (Equations  2.1  &  2.2).  Inclusion  of  a 
fixed  cost  variable  yields  the  following  fixed  cost  model,  which  we  labeled  Equation  1.2: 

C(Q)  =  (F/R)  +  TlQb  (1.2) 

where 

C(Q)  =  cost  to  produce  the  Qth  unit 
Ti  =  first  unit  cost  (model  parameter) 

Q  =  unit  number  (whose  cost  is  to  be  computed) 
b  =  slope  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

R  =  rate  of  production  (quantity  per  time  period  or  lot) 

F  =  fixed  cost  per  lot  (model  parameter) 
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Conceptually  logical,  the  fixed  cost  equation  opens  the  door  for  experimentation 
to  see  if  the  equation  creates  valid  estimates.  Our  research  plans  to  highlight  the 
estimating  capabilities  of  Equation  1.2  and  lobby  that  the  equation  should  become  more 
widely  accepted. 

Dr.  David  Lee’s  Cost  Improvement  Models 

Dr.  Lee  presents  Equation  1.1,  which  we  refer  to  as  the  production  rate 
adjustment  model,  and  which  can  be  seen  below  with  a  description  of  variables. 

C(Q)  =  T]Qb(R/R0)c  (1.1) 

where 

C(Q)  =  cost  to  produce  the  Qth  unit 
Ti  =  first  unit  cost  (model  parameter) 

Q  =  unit  number  (whose  cost  is  to  be  computed) 
b  =  slope  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

R  =  current  production  rate  (number  of  units  produced  in  a  production  period) 
Ro  =  planned  production  rate  (production  rate  prior  to  production  rate  decrease) 
c  =  rate  coefficient  (model  parameter)  =  ln(slope)/ln(2) 

In  The  Cost  Analyst’s  Companion ,  Dr.  Lee  mathematically  derives  the  equations 
from  Chapter  I  we  re-named  Equations  1.1  &  2.1.  These  two  equations  are  the  basis  for 
our  research.  Dr.  Lee  presents  two  theoretical  situations  where  his  two  equations  should 
be  applied  and  will  outperform  any  other  variations  of  the  cost  improvement  curve. 
Equation  1.1  (C(Q)  =  TiQb(R/R0)c)  should  be  applied  when,  “Factors  of  production  can 
change  with  rate,  to  keep  the  facility  operating  at  its  designed  rate”  (Lee,  1997:60).  Dr. 
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Lee  states  that  this  might  only  be  possible  if  production  has  not  yet  begun  and  the 
manufacturer  has  not  yet  constructed  the  plant  (Lee,  1997:60).  Equation  1.2  (C(Q)  = 
(F/R)  +  TiQb)  should  be  applied  when  such  things  as  factory  floor  space,  specialized 
machinery,  and  tooling  cannot  be  easily  changed  without  cost  (Lee,  1997:61).  Dr.  Lee 
concludes  that  the  fixed  cost  equation  (Equation  1.2)  is  more  practical  than  Equation  1.1 
for  actual  programs  facing  changes  in  production  rate.  Dr.  Lee  applies  Equation  1.1  and 
Equation  1.2  each  to  one  set  of  data,  but  does  not  fully  evaluate  them.  His  book  serves 
more  as  a  compilation  of  theoretical  possibilities  rather  than  an  evaluation  of  methods. 
Our  research  will  focus  on  the  evaluation  of  these  two  equations. 

Grouped  together,  Hoffmayer’s  report  (1974),  Moses’  report  (1990),  and  Dr. 

Lee’s  book  (1997)  explain  the  responsiveness  of  production  rate  equations  to  the  major 
cost  drivers.  Most  importantly,  the  reports  explained  how  the  production  rate  adjustment 
models  more  aptly  respond  to  reductions  in  production  lot  quantities.  The  results 
mentioned  above  drove  the  need  for  our  research  and  helped  us  to  limit  and  define  the 
number  of  equations  we  need  to  evaluate. 

Chapter  Summary 

This  chapter  provided  an  extensive  review  of  cost  improvement  models.  Wright 
and  Crawford  developed  the  cumulative  average  model  and  unit  cost  model,  respectively, 
to  explain  the  concept  of  learning.  Learning  describes  the  efficiencies  that  are  gained  as  a 
task  is  executed  repeatedly.  The  laborers  become  better  at  completing  the  tasks  and  the 
unit  cost  of  an  item  decreases  as  more  units  are  produced.  These  two  models  became 
widely  accepted  across  many  disciplines  because  of  their  predictive  capabilities  and 
parsimonious  constructions.  The  first  models  developed  by  Wright  and  Crawford 
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provided  the  concepts  of  learning,  but  did  not  capture  nor  incorporate  all  aspects  of 
production  costs. 

Later  models  built  upon  the  concept  of  learning  to  thus  include  improvements  in 
overall  production.  The  improvements  explain  more  than  the  work  done  by  a  single 
laborer;  they  capture  efficiencies  for  the  entire  production  plant  as  a  unit.  The  uniqueness 
of  products  and  production  plants  provided  impetus  for  expansion  of  the  original  models, 
and  resultantly  many  different  models  surfaced.  In  1974,  Hoffmayer  discovered  the  need 
to  model  the  effects  of  production  rate  (lot  quantity)  changes  on  cost  improvement.  The 
production  rate  model  attempted  to  model  the  findings  of  Hoffmayer’ s  work  and  model 
the  economies  and  diseconomies  of  scale  in  production. 

Moses  evaluated  the  production  rate  equation  and  determined  that  under  certain 
circumstances  the  model  outperforms  the  unit  cost  model  (Crawford’s  model).  Further 
research  into  the  hidden  costs  of  production  discovered  a  conceivable  need  to  include 
fixed  costs  in  the  cost  improvement  model.  Dr.  Lee  recognized  the  need  for  a  fixed  cost 
variable,  as  well  as  for  a  slight  modification  of  the  production  rate  model  to  capture  the 
costs  when  a  program  suffers  changing  lot  quantities.  In  Chapter  III,  we  will  build  on  the 
work  presented  in  Chapter  II  to  evaluate  equations  1.1  and  1.2.  Subsequent  chapters  will 
explain  our  results  and  explore  the  possible  implications  of  our  research. 
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III.  Methodology 


Previous  chapters  discussed  the  need  for  accurate  cost  estimates  and  the  academic 
progression  of  cost  improvement  models.  Through  our  methodology  we  aim  to  discover 
which  cost  improvement  model  should  be  employed  in  cost  estimates  with  decreasing  lot 
sizes.  The  works  detailed  in  Chapter  II  provide  much  of  the  structure  of  our  Monte  Carlo 
simulation,  our  model  creation,  and  our  analysis.  Moses  provides  independent  variables 
affecting  cost,  which  can  be  evaluated  in  conjunction  with  the  cost  estimates  to  determine 
the  variable’s  effect,  if  any,  on  the  estimates.  (Moses,  1990:13).  O.  Douglas  Moses 
identifies  the  predictive  ability  of  the  Crawford  unit  cost  model  and  production  rate 
model,  while  Dr.  David  Lee  explains  the  two  cost  improvement  equations  (Equations  1.1 
and  1.2)  he  determines  to  have  the  most  predictive  ability  when  evaluating  decreases  in 
lot  quantities.  (Lee,  1997:60-61).  We  are  focusing  our  evaluation  on  these  two  equations. 

To  accomplish  our  evaluation,  we  utilize  Microsoft  Excel  with  a  Visual  Basic  for 
Applications  (VBA)  Macro  to  create  the  simulated  cost  data  under  varying  assumptions, 
and  the  Microsoft  Excel  Premium  Solver  Platform  add-in  to  optimize  each  model.  While 
Moses  recognized  that  his  cost  data  simulation  limited  the  usefulness  of  the  research,  we 
intend  to  demonstrate  how  the  reasonable,  yet  exhaustive,  assumptions  of  our  simulations 
strengthen  our  research.  (Moses,  1990:30).  Microsoft  Excel  affords  us  the  opportunity  to 
provide  a  more  robust  Monte  Carlo  simulation;  thus  we  have  the  responsibility  to  provide 
an  extensive  simulation.  As  more  advanced  tools  become  available  to  analysts,  the 
improved  technology  furnishes  the  ability  to  challenge  the  status  quo  of  cost  estimation, 
our  goal  with  this  research  effort. 
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Basic  Evaluation  Structure 


The  basic  structure  of  our  evaluation  method  can  be  seen  in  Figure  3.1. 


Determine 

Independent  Variable 
levels  for  Monte  Carlo 
simulation 


Use  lot  quantities  and  lot 
costs  to  fit  the  production 
rate  adjustment  equation. 

Inctimota  mnrlal 


Run  simulation  to 
develop  “historical” 
cost  data 


Use  lot  quantities  and  lot 
costs  to  fit  the  fixed  cost 
equation.  Estimate 
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Our  evaluation  structure  is  simple  and  easy  to  follow.  Following  this  flow  chart 
allows  us  to  preserve  the  same  structure  as  we  repeat  the  process  under  different 
simulation  assumptions.  Even  though  we  assembled  a  simple  structure,  Microsoft  Excel 
allows  us  to  evaluate  the  cost  improvement  equations  under  complicated  assumptions. 
Our  methodology  focuses  on  creating  many  possible  cost  profiles  and  comparing  each  of 
the  cost  improvement  models  on  their  predictive  capability  with  the  same  cost  data. 

The  Cost  Generating  Functions 

Our  cost  generating  functions  mimic  similar  patterns  in  historical  data.  We  model 
samples  from  our  simulated  production  costs  against  the  basic  learning  curve  shown  in 
Chapter  II  as  well  as  historical  costs  to  ensure  similarities.  Creating  a  cost  function 
requires  some  guesswork  because  if  producers  knew  their  true  cost  generating  functions, 
then  the  need  for  analysts  would  not  exist  (Moses,  1990:8).  We  must  do  our  best  to 
recreate  patterns  because  we  do  not  know  true  cost  functions  for  each  individual 
situation.  The  basic  production  cost  structure  of  any  particular  item  consists  of  a  fixed- 
cost  portion  and  a  variable-cost  portion.  As  mentioned  in  Chapter  II,  fixed  costs  did  not 
originally  factor  into  cost  improvement  models,  but  as  research  grew,  analysts 
acknowledged  fixed  cost  influence  on  cost  improvement  models.  We  used  two  different 
cost  generating  functions.  We  determined  the  need  for  two  cost  generating  functions  to 
avoid  favoring  either  equation  thus  biasing  the  analytical  process,  and  the  necessity  to 
create  different  cost  scenarios  in  order  to  provide  a  thorough  analysis.  We  sampled 
construction  from  the  cost  improvement  research  accomplished  by  Avinger  (1987), 

Moses  (1990),  and  Thomas  (1975)  to  determine  our  cost  functions.  The  two  cost 
functions  are  simulated  and  modeled  separately;  the  simulation  results  for  each  cost 
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function  will  be  discussed  in  Chapter  IV  of  our  research.  To  account  for  the  inherent 
fixed  cost  in  every  unit,  the  first  cost  generating  function  contains  a  fixed  cost  portion: 

UC  =  VCiQb  +  (FC/R)  +  s  (3.1) 

where 

UC  =  unit  cost 

VCi  =  variable  cost  of  the  first  unit  of  production 
Q  =  unit  number  (cumulative  over  production  life) 
b  =  variable  cost  learning  rate  =  log(learning  slope)/log(2) 

FC  =  fixed  cost  for  the  production  period,  which  is  the  same  for  every 
production  period 

R  =  production  rate/lot  quantity 
s  =  error  term 

Adding  a  learning  rate  to  the  equations  validates  the  influence  of  the  actual  learning  that 
occurs  through  repetitive  production.  We  assume  there  are  no  breaks  in  production 
affecting  the  learning  rate,  such  that  each  production  period  maintains  the  same  learning 
rate  in  a  continuous  calculation.  We  chose  to  keep  the  fixed  costs  for  each  lot  equal  to 
capture  the  contractual  obligations  of  the  manufacturers  and  to  acknowledge  their 
inability  to  change  fixed  costs  as  production  rates  vary.  The  producers  are  tied  to 
manufacturing  plant  size,  labor  training  costs,  administrative  costs,  raw  materials  orders, 
and  other  fixed  costs  that  cannot  be  avoided.  Normally,  as  production  increases, 
manufacturers  can  capitalize  on  quantity  discounts  and  thus  spread  fixed  costs  over  more 
units.  When  the  production  rate  suffers  an  unanticipated  decrease,  the  short  run  fixed 
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costs  cannot  be  modified.  These  large  fixed  costs  can  only  be  spread  across  the  smaller 
number  of  units,  meaning  that  each  unit  bears  a  larger  portion  of  the  fixed  cost.  The 
fixed  cost  represents  the  economies  or  diseconomies  of  scale  within  a  production  cycle. 

Because  the  fixed  cost,  cost  improvement  equation  favors  Equation  3.1  through 
the  fixed  cost  variable,  we  also  created  a  more  generic  cost  function.  The  second  cost¬ 
generating  function  excludes  a  fixed  cost  variable. 

UC  =  VCiQb  +  s  (3.2) 

where 

UC  =  unit  cost 

VCi  =  variable  cost  of  the  first  unit  of  production 
Q  =  unit  number  (cumulative  over  production  life) 
b  =  variable  cost  learning  rate  =  log(learning  slope)/log(2) 
s  =  error  term 

Equation  3.2  does  not  include  the  fixed  cost  variable;  because  the  learning  rate  can  only 
reduce  the  unit  cost  to  a  certain  level,  a  stable  cost  per  unit  that  cannot  be  eliminated 
exists.  Equation  3.2  is  identical  to  the  original  learning  curve  equations  developed  by 
Crawford.  This  equation  captures  the  fundamental  aspects  of  producing  a  good  and 
allows  us  to  vary  our  independent  variables  to  create  unique  cost  profiles. 

Independent  Variables 

Multiple  independent  variables  affect  cost.  Monte  Carlo  simulation  allows  us  to 
vary  multiple  independent  variables  simultaneously  to  build  cost  profiles.  The 
independent  variables  work  within  the  boundaries  of  the  cost  generating  functions  to 
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mimic  historical  costs  and  to  truly  test  our  cost  improvement  models.  We  chose  the 
independent  variables  to  be  inserted  for  analysis  into  our  equations:  number  of  lots,  lot 
quantity,  cumulative  quantity,  unit  one  cost,  variable  cost  learning  rate,  fixed  cost  burden, 
production  rate  decrease,  and  noise.  In  the  following  subsections  we  provide 
explanations  for  each  of  these  variables  and  explain  how  we  created  our  simulations. 
Through  utilization  of  independent  variables,  we  discover  statistical  patterns  and 
indications  of  reliable  forecasting.  Following  the  variable  descriptions,  Table  3.1  shows 
the  distributions  and  parameters  for  each  variable  in  the  Monte  Carlo  simulation  for  each 
cost-generating  function. 

Number  of  Production  Lots  ( History ) 

Number  of  production  lots  refers  to  how  many  production  lots  will  be  used  as 
"historical"  data.  With  lower  numbers  of  production  lots,  fewer  observations  are 
available  for  modeling  possibly  affecting  how  the  two  cost  improvement  models  perform. 
It  is  possible  that  a  given  cost  improvement  model  estimates  extremely  effectively  with 
more  data  points,  but  drastically  underperforms  when  a  significantly  smaller  number  of 
data  points  are  available.  Models  could  also  be  affected  by  the  amount  of  historical  data, 
thereby  overestimating  or  underestimating  consistently  based  on  a  certain  amount  of  data 
history  tainting  the  models'  accuracy  and  validity. 

Our  Monte  Carlo  simulation  uses  uniform  discrete  distribution  ranging  from  three 
production  lots  to  ten  production  lots.  The  uniform  distribution  assigns  equal 
probabilities  to  each  value  within  that  range  and  randomly  assigns  an  integer  value. 

Based  on  previous  academic  research,  we  chose  these  values  because  they  represent  the 
typical  amount  of  data  available  for  cost  improvement  modeling.  Values  below  three  do 
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not  provide  enough  data  for  cost  analysts  to  model,  and  values  above  ten  do  not  typically 
occur  because  production  does  not  usually  span  over  ten  periods.  Our  range  parallels  the 
minimum  and  maximum  values  of  Moses’  research,  but  he  did  not  account  for  all  values 
within  that  range  as  we  do  (Moses:  13). 

Lot  Quantity 

The  historical  lot  quantities  represent  full  rate  production  levels.  We  do  not 
model  ramped  up  production  because  the  specific  situations  we  address  occur  during  the 
later  life  of  individual  programs.  Also,  in  the  early  stages  of  a  program,  a  producer 
prototypes  products  and  refines  the  manufacturing  process  as  needed;  we  do  not  want  to 
account  for  the  adjustments  made  before  production  stabilizes.  The  simulation  adds 
variations  to  each  production  period  because  production  levels  are  commonly  unstable 
from  period  to  period. 

Lot  1  quantities  range  from  a  starting  value  of  15  to  an  upper  ceiling  of  60.  The 
integer  values  within  this  range  are  uniformly  distributed,  which  again  means  that  each 
value  holds  an  equal  probability  of  occurrence  through  random  generation.  The  range 
accounts  for  the  need  to  model  programs  with  high  productivity  as  well  as  programs  with 
lower  product  output.  Our  range  is  somewhat  arbitrary  demonstrating  potentially 
extreme  maximum  and  minimum  values,  but  it  is  arguably  representative  of  possible 
scenarios,  thus  offering  a  truer  test  of  our  estimation  models. 

Each  subsequent  lot  uses  the  previous  lot  quantity  and  a  triangular  distribution 
with  that  value  as  the  mode  (center)  value.  This  ensures  that  each  production  run  builds 
on  information  within  that  scenario.  Lot  quantities  are  critical  to  data  simulation  but  are 
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not  used  during  analysis.  The  lot  quantity  data  is  summed  to  produce  the  cumulative 
quantity,  which  can  be  analyzed  against  the  cost  improvement  model. 

Cumulative  Quantity 

Measuring  cumulative  quantity  of  the  production  line  allows  us  to  observe  how 
the  equations  behave  with  different  program  sizes.  We  aim  to  uncover  any  pattern  or 
statistical  significance  associated  with  the  models  and  the  models'  abilities  to  predict 
based  on  program  size.  The  cumulative  quantity  builds  from  the  number  of  lots  and  the 
lot  quantity  data. 

Unit  One  Cost  (UCi) 

The  cost  of  the  first  unit  of  production  represents  how  much  of  the  total  cost  of 
the  first  unit  of  production  can  be  attributed  to  variable  cost  and  is  inherently  affected  by 
predictable  learning  on  the  production  line.  The  first  unit  cost  determines  the  starting 
point  for  subsequent  cost  calculations  of  learning  rate  and  fixed  cost  burden.  Fixed  cost 
is  added  as  a  percentage  of  total  cost  and  is  not  affected  by  learning.  To  capture  all 
relevant  cost  structures  we  created  an  extensive  range  of  simulated  values. 

We  simulated  the  first  unit  variable  cost  through  a  uniform  continuous 
distribution  between  10,000  and  1,000,000  dollars.  The  value  can  be  interpreted  as 
thousands  of  dollars  to  more  accurately  relate  to  the  flyaway  cost  of  an  airframe,  but  this 
inclusion  does  not  change  the  analysis.  Every  value  within  that  range  has  an  equal 
probability  of  becoming  the  first  unit  cost  as  we  process  individual  iterations  of  our 
simulation.  We  wanted  ensure  with  certainty  that  we  tested  the  predictive  ability  of  the 
two  cost  improvement  models  over  a  range  of  large  and  small  values.  The  smaller  values 
represent  low  cost  items  while  the  large  cost  items  depict  major  manufacturing  items 
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and/or  total  production.  Though  we  cannot  model  every  value  within  that  range,  we 
succeed  in  creating  a  mixture  of  values  representative  of  varying  cost  profiles.  The 
simulation  applies  a  learning  rate  to  each  of  these  cost  profiles  to  create  values  over  a 
period  of  time. 

Variable  Cost  Learning  Rate  (LR) 

Learning  Rate  describes  the  actual  amount  of  learning  taking  place  on  the 
production  line.  As  laborers  repeat  the  same  processes  over  a  certain  time  period,  the 
laborers  achieve  more  efficiency  and  complete  the  same  amount  of  work  in  a  lesser 
amount  of  time  than  when  they  initiated  the  production  process.  Due  to  the  progressive 
manufacturing  efficiency,  the  unit  variable  cost  will  decrease  by  a  constant  percentage  as 
the  number  of  units  produced  doubles.  The  variable  cost  learning  rate  is  the  same  as  the 
learning  rate  described  in  Chapter  II,  and  further  examples  can  be  read  in  Chapter  II  of 
this  research. 

We  first  chose  to  use  a  triangular  distribution  to  model  the  learning  rate  with  the 
minimum  expected  value  set  as  75  percent  learning,  the  maximum  value  set  as  95  percent 
learning,  and  the  most  likely  learning  rate  as  80  percent.  The  values  are  based  on  similar 
academic  research  evaluating  cost  improvement  models  (Moses,  1990:8;  Avinger, 
1987:18).  Choosing  a  triangular  distribution  allows  us  to  restrict  learning  rates  to  values 
between  75  percent  and  95  percent  while  acknowledging  that  those  minimum  and 
maximum  values  have  a  low  probability  of  occurrence.  A  normal  distribution  will 
model  similar  principles,  but  will  also  allow  for  the  occurrence  of  extremely  high  and  low 
values  not  consistent  with  actual  production.  For  example,  if  the  value  of  100  percent 
learning  is  randomly  selected  through  a  normal  distribution,  then  zero  learning  will  occur 
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throughout  the  life-cycle  of  that  production  line.  Only  under  the  most  rare  of 
circumstances  will  this  situation  exist  and  throughout  multiple  production  lots  it  is 
unlikely  the  situation  will  remain  consistent. 

Based  upon  the  input  from  our  in-person  conversations  with  Mr.  Ken  Birkofer 
and  Mr.  Doug  Mangen,  cost  analysts  with  the  F-22  Program,  we  constructed  multiple 
scenarios  for  the  learning  rate.  In  their  experience,  the  learning  rate  for  larger  production 
scenarios  stays  between  85  percent  and  95  percent,  while  assembly  line  production  is 
usually  75  percent  to  85  percent.  Our  first  distribution  spans  over  the  entire  range,  but  we 
also  use  distributions  to  model  the  two  other  levels  independently.  By  limiting  the 
learning  rates  to  these  values,  we  can  detect  if  the  different  levels  affect  the  predictive 
abilities  of  Equations  1.1  and  1.2.  As  mentioned  previously,  variable  costs  are  the  only 
costs  affected  by  learning,  but  variable  costs  are  not  the  only  costs  composing  the  cost 
structure  of  a  unit  and  an  entire  production  period.  Fixed  costs  play  a  role  in  determining 
the  cost  of  future  lots. 

Fixed  Cost  Burden  (FC) 

Fixed  cost  burden  represents  the  percentage  of  total  cost  not  affected  by  learning 
and  held  constant  throughout  all  production  periods.  With  higher  production  rates  the 
production  plant  gains  efficiencies  and  fixed  costs  can  be  distributed  across  more  units  to 
lower  the  cost  per  unit.  When  a  plant  experiences  lower  production  rates,  the  fixed  costs 
become  a  higher  percentage  of  unit  costs  and  efficiencies  are  lost.  In  our  specific 
situation,  the  unanticipated  production  decreases  do  not  give  the  manufacturer  the 
opportunity  to  change  the  already  established  fixed  cost  structure.  The  manufacturer 
cannot  sell  a  portion  of  the  manufacturing  plant,  cut  contractual  agreements  for  supply 
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purchases,  reduce  storage  space,  un-train  laborers,  nor  reduce  fixed  costs  in  any  other 


way. 

We  assigned  discrete,  equal  distribution  to  fixed  cost  burdens  of  10  percent,  20 
percent,  and  30  percent.  The  simulation  uses  the  variable  costs  of  the  first  production  lot 
to  create  a  total  cost  profile  where  one  of  these  values  is  used  to  represent  the  fixed  cost 
of  the  lot.  The  value  calculated  from  the  first  lot  becomes  a  consistent  fixed  cost  for  each 
production  period,  where  the  amount  assigned  to  each  unit  depends  on  the  production 
rate.  The  proposed  cost  improvement  models  claim  that  there  are  hidden  costs  inherent 
to  production  that  cannot  be  avoided  and  that  need  to  be  modeled  in  the  cost 
improvement  model.  The  fixed  cost  burden  provides  a  value  that  is  consistent  throughout 
production  lots,  that  can  be  affected  by  the  production  rate,  and  that  should  be  captured 
by  the  cost  improvement  models.  For  the  future  production  lot  that  will  be  estimated,  the 
fixed  cost  becomes  crucial  as  the  production  period  suffers  decreases  in  production.  The 
production  rate  decrease  determines  how  many  units  will  bear  the  burden  of  the  fixed 
cost. 

Production  Rate  Decrease  for  the  Estimate  Lot  (PR) 

The  main  goal  of  our  research  is  to  demonstrate  how  the  unanticipated  cut  in 
production  levels  will  affect  the  cost  improvement  model’s  ability  to  forecast.  Thus,  the 
future  production  lot  that  needs  to  be  estimated  will  display  a  production  decrease 
affecting  actual  cost.  Production  decreases  represent  any  program  changes  whether  need- 
based  or  funding-based  that  can  lead  to  fewer  units  being  required  and  therefore  produced 
than  were  previously  anticipated  and  projected.  Because  our  simulated  data  does  not 
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belong  to  a  program  with  a  history  of  projected  production  lot  sizes,  the  final  historical 
lot  size  serves  as  the  future  production  level  experiencing  a  decrease. 

Our  Monte  Carlo  simulation  represents  production  rate  decreases  with  a  discrete, 
equal  distribution  of  the  values  25  percent,  50  percent,  and  75  percent.  For  example,  if 
the  final  historical  production  level  is  100  units  and  suffers  a  25  percent  production  rate 
decrease,  then  the  future  production  rate  will  be  75  units.  There  is  no  pattern  for  the 
amount  of  decrease  programs  will  face.  The  amount  of  decrease  depends  on  the 
economic  and  social  climate  of  the  time  and  on  the  nature  of  the  program.  By  choosing 
these  values  we  can  simulate  low,  medium,  and  high  production  decreases  and  thereby 
measure  how  the  cost  improvement  models  predict  future  costs.  The  cost  information  for 
future  production  lots  is  simulated  simultaneously  with  the  historical  production  lots  so 
the  model  estimations  can  be  evaluated  against  ‘true’  costs. 

Noise 

Noise  represents  the  unpredictable  natures  of  production  situations  and  estimating 
costs.  Adding  the  noise  variable  presents  the  best  option  to  account  for  the  unknown 
events  of  a  production  run.  Noise  represents  any  unforeseen  and  even  unknown  events 
that  occur  at  any  time  during  production. 

We  create  noise  based  on  the  following  function: 

Noise  =  UC  *  e  (3.3) 

where 

UC  =  unit  cost  for  that  particular  unit 

e  =  randomly  generated  percentage  based  on  our  distribution 
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Noise  affects  each  unit  cost  calculation,  and  in  our  simulation  noise  can  be  any 
value  between  -0.05  percent  and  15.0  percent  of  that  unit  cost.  The  values  are 
represented  by  uniform  continuous  distribution  where  every  value  has  an  equal 
probability  of  occurrence  through  random  number  generation.  Though  normal 
production  cycles  may  not  fit  this  noise  distribution  where  a  value  of  15.0  percent 
appears  as  often  as  a  value  closer  to  zero,  we  aim  to  generate  a  high  number  of 
possibilities  to  observe  how  each  of  the  models  reacts.  This  range  offers  a  sample  with 
more  extreme  values  to  observe,  while  also  providing  lower,  more  conservative,  values 
for  evaluation.  We  did  not  limit  ourselves  to  one  possible  distribution  of  noise;  we  also 
simulated  a  normal  distribution  with  a  mean  of  zero  and  a  standard  deviation  of  7.5 
percent.  By  using  different  distributions  we  can  see  if  and  how  noise  affects  the 
predictive  abilities  of  the  equations. 

Table  of  Simulated  Values 

Table  3.1  only  shows  the  first  set  of  assumptions;  we  modeled  assumptions  for 
learning  rate  and  noise  at  different  levels  to  explore  any  changes  in  the  results. 
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Table  3.1.  Independent  Variable  Descriptions  for  Monte  Carlo  Simulation  (1st  Set) 


Independent  Variable 

Distribution 

Range 

Number  of  Production  Lots 

Uniform  Distribution 

Min:  3  Lots 

Max:  10  Lots 

Lot  Quantity 

Lot  1:  Uniform 

Distribution 

Other  Lots:  Triangular 
Distribution 

Lot  1  Min:  15  Units 

Lot  1  Max:  60  Units 

Other  Lots  Mode:  Previous 
Lot’s  Number  of  Units 

Other  Lots  Min:  Mode  -  5 
Other  Lots  Max:  Mode  +5 

Cumulative  Quantity 

No  Distribution 

Summation  of  Lot 

Quantities 

Unit  One  Cost 

Uniform  Distribution 

Min:  $10,000 

Max:  $1,000,000 

Variable  Cost  Learning 

Rate 

Triangular  Distribution 

Min:  75.0% 

Mode:  85.0% 

Max:  95.0% 

Fixed  Cost  Burden 

Discrete  Distribution 

Possible  Outcomes: 

10.0%,  20.0%,  30.0%,  and 
40%  of  Total  Cost 

Production  Rate  Decrease 

Discrete  Distribution 

Possible  Outcomes:  25%, 
50%  and  75%  Decreases 

Noise 

Uniform  Distribution 

Min:  -5.0% 

Max:  15.0% 

Monte  Carlo  Simulation  and  Model  Creation 

Monte  Carlo  simulation  provides  us  with  the  ability  to  model  thousands  of 
different  cost  profiles  and  then  to  evaluate  the  results.  Through  random  number 
generation  within  the  distribution  guidelines  for  each  variable  previously  mentioned,  we 
create  costs  that  exhibit  similar  patterns  to  historical  production  costs. 

We  use  Microsoft  Excel  Visual  Basic  for  Applications  and  Premium  Solver 
Platform  from  Frontline  Systems  to  create  a  workbook  that  allows  us  to  create  historical 
production  costs,  to  model  the  parameters  of  each  cost  improvement  equation,  to  use  the 
model  parameters  to  estimate  the  next  production  lot,  to  simulate  the  cost  of  the  next 
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production  lot,  to  measure  the  prediction  errors,  and  to  store  simulation  and  model 
information. 

Visual  Basic  for  Applications  (VBA) 

The  code  creates  a  program  that  runs  on  the  command  of  the  user.  Based  on  the 
structured  Microsoft  Excel  worksheets,  the  VBA  code  creates  all  of  our  essential  data  for 
more  accurate  evaluation  of  the  cost  improvement  models.  The  code  runs  the  simulation, 
stores  snapshots  of  the  simulated  conditions  (levels  for  independent  variables),  runs 
Solver  to  find  model  parameters,  stores  the  model  parameters,  and  stores  the  estimated 
values.  The  process  runs  in  an  integrated,  fluid  motion  where  the  user  only  needs  to  input 
the  number  of  iterations  that  need  to  be  calculated.  Microsoft  Excel  automatically 
calculates  information  while  the  VBA  runs  the  simulation,  and  the  Solver  function  finds 
the  optimal  model  parameters  for  estimation. 

Model  Creation 

Model  creation  follows  the  guidance  offered  in  The  Air  Force  Cost  Analysis 
Handbook.  In  our  study,  models  are  formulated  from  historical  data  points  with  no 
discernible  knowledge  of  the  inter- workings  of  the  program.  Within  our  research,  the 
only  elements  of  information  available  are  unit  cost  and  lot  cost  for  production.  Only  one 
data  point  is  derived  from  each  set  of  lot  data;  the  points  consist  of  a  lot  midpoint  unit 
number  and  an  average  unit  cost  per  lot.  We  calculate  the  lot  midpoint  using  a  heuristic 
for  the  unit  number.  “A  lot  midpoint  is  the  unit  number  (not  necessarily  a  whole  number) 
that  corresponds  with  the  average  unit  cost  for  a  given  lot  under  the  Unit  curve 
formulation”  (Handbook:  8-22).  Lot  midpoints  can  only  be  used  in  the  unit  cost  formula 
of  the  cost  improvement  model,  which  is  consistent  with  our  research  because  production 
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rate  adjustments  must  be  based  on  unit  cost  methods  (Handbook:  8-22,  8-32).  The  lot 
midpoint  heuristic  used  is: 

(LS+LE+2+  a JLS*LE ) 

Lot  Midpoint  = - - -  (3.3) 

where 

LS  =  cumulative  production  number  of  the  first  unit  in  the  lot 
LE  =  cumulative  production  number  of  the  last  unit  in  the  lot 

The  lot  midpoint  and  average  unit  cost  create  the  historical  data  points  to  find  our  model 
parameters.  With  those  data  points  we  use  Premium  Solver  Platform  to  find  the 
statistically  correct  model  parameter  for  estimation. 

Premium  Solver  Platform 

Microsoft  Excel  offers  a  Solver  add-in  called  Premium  Solver  Platform.  Solver 
operates  as  an  optimization  tool  to  find  the  value  of  parameters  within  a  model  that  meets 
a  goal  or  condition.  Frontline  System  states  on  their  website  that  Premium  Solver 
Platform  “solves  every  type  and  size  of  problem,  using  built-in  and  plug-in  Solver 
Engines”  (www.solver.com).  Premium  Solver  Platform  is  “a  unique  combination  of 
genetic  algorithms  and  classical  nonlinear  optimization  methods”  (www.solver.com).  As 


a  reminder,  our  four  equations  for  modeling  are: 

The  Production  Rate  Adjustment  Model:  C(Q)  =  TiQb(R/Ro)c  (1.1) 

The  Fixed  Cost  Model:  C(Q)  =  (F/R)  +  TjQb  (1.2) 

The  Crawford  Unit  Cost  Model:  C(Q)  =  T 1 0b  (2.2) 

The  Production  Rate  Model:  C(Q)  =  Ti<2b  Rc  (2.4) 
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Three  of  our  cost  improvement  models  are  nonlinear  (Equations  1.1,  1.2  and  2.4) 
and  require  the  more  robust  solver  to  calculate  the  model  parameters.  More  specifically, 
we  need  to  be  certain  that  Solver  can  calculate  the  global  minimums  while  avoiding  the 
local  minimum  values.  To  combat  this  problem,  Premium  Solver  Platform  developed  a 
multi-start  function  to  identify  global  solutions.  The  multi-start  function  “can  be 
automatically  run  many  times  from  judiciously  chosen  starting  points,  and  the  best 
solution  found  will  be  returned  as  the  optimal  solution.  [. . .]  multi-start  methods  will 
converge  in  probability  to  the  globally  optimal  solution”  (www.solver.com).  With  the 
use  of  Premium  Solver  Platform  we  calculate  the  model  parameters  for  each  model  on 
each  set  of  production  costs. 

Solver  requires  that  a  condition  be  optimized  to  find  the  model  parameters.  We 
chose  to  calculate  the  Sum  of  Squared  Error  (SSE)  for  each  model  fitting  the  data,  and  to 
minimize  SSE  to  define  the  optimum  model  parameters.  SSE  identifies  the  error  for  each 
data  point,  squares  the  error,  and  then  adds  the  resulting  product  to  the  values  of  other 
data  points.  By  choosing  different  values  for  the  model  parameters,  the  SSE  can  be 
minimized  thus  reducing  the  fitting  error.  For  the  production  rate  adjustment  equation 
(Equation  1.1)  cost  improvement  model  we  optimize  the  values  of  b,  c,  and  Ti.  For  the 
fixed  cost  equation,  (Equation  1.2)  cost  improvement  model  we  optimize  b,  F,  and  Ti. 
The  values  of  T \  and  F  are  not  the  true  Unit  one  values  or  fixed  costs  amount,  but  rather 
Ti  and  F  are  theoretical  parameters  established  to  create  the  best  estimating  model. 

Solver  converged,  based  on  probability,  to  the  optimum  value  of  these  model  parameters 
in  order  to  minimize  SSE.  Premium  Solver  Platform  generates  outputs  based  upon  the 
Solver  iterations’  simulated  data,  delineating  the  steps  of  the  program. 
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Limitations 


Modeling  every  possible  production  scenario  is  impossible  and  an  obvious 
limitation.  We  attempt  to  include  all  variables  affecting  costs  along  with  making 
reasonable  selections  of  values  for  the  variables,  while  heavily  relying  on  the  power  and 
accuracy  of  Premium  Solver  Platform  to  calculate  the  correct  model  parameters.  All 
indicators  lead  us  to  believe  that  the  program  converges  on  the  best  possible  solution,  but 
we  do  not  have  the  means  to  personally  test  and  analyze  every  model.  Any  anomalies 
within  the  data  will  be  researched  and  explained  in  the  analysis.  The  assumptions  for 
accuracy  of  fit,  equal  variance,  and  independence  will  not  be  evaluated  with  our  results. 
We  focus  on  the  predictive  ability  of  each  of  the  models;  testing  the  assumptions  of 
thousands  of  model  iterations  is  simply  not  feasible.  We  have  done  our  best  to  create 
sound  methodology  for  our  Monte  Carlo  simulation  and  model  creation,  but  acknowledge 
that  unknown  or  unrecognized  factors  can  impact  the  research. 

Model  Evaluation  with  Historical  Data 

The  Air  Force  Cost  Analysis  Agency  provided  unit  recurring  flyaway  data  for  a 
number  of  aircraft  platforms.  Summary  information  represents  data  that  was  collected, 
normalized  and  analyzed  by  RAND  Corporation.  Any  anomalies  have  been  explained 
and  removed  from  the  data  to  identify  a  cost  structure  that  can  be  modeled.  The 
summary  data  provides  lot  quantity  data  and  average  unit  costs  for  each  lot.  We  will 
model  these  observations  using  the  same  methods  mentioned  above  for  our  simulated 
data.  We  will  use  lot  midpoint  and  average  unit  cost  data  points  to  run  Solver  Premium 
Platform  and  to  solve  for  model  parameters.  Future  costs  to  measure  the  predictive 
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accuracy  are  unknown,  thus  we  will  withhold  the  data  point  where  a  production  decrease 
takes  place  and  will  use  that  as  the  ‘future’  data  point. 

Measures  of  Error 

Mean  Percentage  Error  (MPE) 

MPE  is  calculated  through  the  following  equation: 

Actual  Cost-Estimated,  Cost 

MPE  =  - 100  (3.4) 

Actual  Cost 

We  used  MPE  to  measure  the  fitting  error  as  well  as  the  forecast  error.  Each  data  point’s 
MPE  is  calculated  separately;  the  MPE  values  are  then  averaged  to  reach  a  final  value. 
The  value  illustrates  the  bias  of  the  model  by  the  positivity  or  negativity  of  the  final  MPE 
value.  A  positive  value  indicates  that  the  model  underestimates  the  data,  while  a  negative 
value  signals  overestimation.  The  fitting  error  illustrates  how  well  the  model  fits  the 
historical  data.  While  MPE  can  be  used  as  an  indication  of  accurate  forecasting  by  the 
model,  the  MPE  does  not  guarantee  that  the  model  will  forecast  precisely. 

We  also  calculate  the  MPE  of  the  forecast  to  compare  that  value  to  the  fitted 
MPE.  Large  differences  in  the  value  show  that  even  though  the  model  fits  the  historical 
data  well,  the  model  is  no  indication  of  future  costs. 

Mean  Absolute  Percentage  Error  (MAPE) 

Mean  Absolute  Percentage  Error  takes  the  absolute  value  of  each  MPE  and  sums 
those  absolute  values.  MAPE  reveals  the  extent  that  the  fit  and  forecast  vary  from  the 
actual  value  without  taking  into  account  whether  the  equations  overestimate  or 
underestimate.  Again,  the  MAPE  fitting  the  historical  data  does  not  guarantee  the  model 
will  provide  an  accurate  forecast,  but  the  MAPE  can  be  used  as  an  indication  of  the 
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predictive  ability  of  the  model.  Measuring  the  difference  between  the  fit  error  and  the 
forecast  error  provides  insight  into  the  validity  of  the  model  and  the  model’s  ability  to 
predict  future  values. 

We  use  SSE,  MPE,  and  MAPE  to  reveal  information  about  each  of  the  models. 
Evaluating  these  values  with  the  independent  variable  values  demonstrates  how  well  the 
models  are  forecasting  and  which  variables  affect  the  individual  model’s  ability  to 
perform. 

Chapter  Summary 

In  this  chapter  we  built  upon  the  literature  reviewed  in  Chapter  II  to  develop  our 
methods  for  evaluating  Equations  1.1  and  1.2.  We  will  be  using  Monte  Carlo  simulation 
to  create  production  costs  that  can  then  be  modeled.  In  our  Monte  Carlo  simulation  we 
will  vary  eight  different  independent  variables  to  create  an  all-inclusive  set  of  possible 
cost  profiles.  Number  of  lots,  lot  quantity,  cumulative  quantity,  unit  one  cost,  variable 
learning  rate,  fixed  cost  burden,  production  rate  decrease,  and  noise  will  all  be  varied  in 
our  simulation.  The  simulated  data  will  mimic  normalized  historical  data  used  for  cost 
estimating.  For  each  set  of  production  costs  we  will  model  the  production  rate 
adjustment  equation  (Equation  1.1)  and  the  fixed  cost  model  (Equation  1.2)  using  the 
Microsoft  Excel  add-in  Premium  Solver  Platform.  The  model  parameters  discerned  using 
Solver  will  be  implemented  to  predict  a  future  lot,  and  the  error  of  that  prediction  will  be 
measured.  Using  historical  data  provided  by  AFCAA,  we  intend  to  verify  our  findings 
and  to  determine  which  equation  better  predicts  future  costs. 

Chapter  IV  shows  the  results  of  our  Monte  Carlo  simulation  and  the  model 
estimation  for  the  simulated  and  historical  data.  We  introduce  the  measures  of  error 
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results  for  each  simulation  to  determine  which  variables  affect  each  model’s  forecast. 


Chapter  IV  provides  a  summary  of  our  analysis,  while  Chapter  V  highlights  the  most 
significant  aspects  of  our  research  and  the  potential  cost  analysis  policy  implications. 


50 


IV.  Results  and  Discussion 


Previous  chapters  included  the  purpose  of  our  research,  an  extensive  literature 
review  on  cost  improvement  curves,  and  our  methodology  for  our  cost  improvement 
models.  In  this  chapter,  we  focus  on  the  results  of  our  Monte  Carlo  simulation,  our  cost 
improvement  model  creation,  and  our  model  evaluation.  We  display  a  random  example 
from  each  of  our  simulated  production  functions  and  compare  individual  production 
profiles  to  historical  cost  profiles  for  validity.  Our  analysis  of  the  Monte  Carlo 
simulation  datasets  includes  measures  of  error,  accuracy  plots,  statistical  analysis,  and 
patterns  revealed  within  the  models.  Our  results  illustrate  that  the  production  rate  cost 
improvement  model  outperforms  the  other  cost  improvement  models  overall.  Originally 
we  hypothesized  that  Equations  1.1  and  1.2  would  furnish  better  results  than  would  the 
other  models,  but  this  did  not  prove  to  be  true.  For  this  reason  we  include  the  results  of 
the  Crawford  unit  cost  model  and  the  production  rate  model  when  applicable  in  this 
chapter.  The  analysis  also  demonstrates  that  fixed  cost  model  produces  statistically  equal 
forecasts  when  production  cycles  possess  high  fixed  costs  and  suffers  losses  of  efficiency 
when  production  decreases  occur.  We  validate  the  performance  of  the  fixed  cost  model 
with  historical  data  where  the  fixed  cost  model  consistently  provides  more  accurate 
estimates  than  do  the  other  models.  Our  evaluation  of  the  cost  improvement  models 
includes  estimations  of  normalized  historical  data;  the  results  remain  consistent  with  our 
findings. 
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Monte  Carlo  Simulation  Results 


Our  cost  generation  functions  produce  reasonable  data  for  our  analysis.  Patterns 
found  in  historical  data  provided  by  the  Air  Force  Cost  Analysis  Agency  fuel  our 
simulations  as  mentioned  in  Chapter  III,  substantiating  the  validity  of  our  results.  The 
following  subsections  elucidate  random  examples  from  each  of  our  cost  functions. 

Production  Cost  Simulation:  Fixed  Cost  Function 

The  below  dataset  displayed  in  Figure  4.1  and  Table  4.1  represents  a  random 
production  run  from  our  simulation.  We  cannot  individually  plot  each  of  the  thousands 
of  dataset  simulations  from  our  research,  but  our  example  illustrates  the  basic  structure 
evident  throughout  our  research.  The  thin  black  line  represents  each  unit  cost  throughout 
production,  while  the  thick  black  line  depicts  the  lot  midpoints  and  the  average  unit  cost 
per  lot  utilized  for  model  creation. 


600,000 


0  -I - , - , - , - , - , - , - , 

0  50  100  150  200  250  300  350 

Unit  Number 


Unit  Cost  ^^—Lot  Midpoint/ Avg  Unit  Cost 


Figure  4.1:  Fixed  Cost  Production  Simulation  Example 
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Figure  4. 1  tracks  the  path  of  a  traditional  learning  curve  production,  but  breaks 
that  pattern  in  the  final  lot  where  the  production  rate  decreases.  The  production  rate 
decrease  triggers  the  average  unit  cost  for  that  production  lot  to  increase  because  each 
unit  must  shoulder  a  larger  portion  of  the  fixed  cost.  The  change  in  the  cost  represents 
our  hypothesized  loss  of  efficiency  associated  with  production  decreases.  Appendix  A 
manifests  this  dataset  in  its  entirety.  Table  4.1  presents  the  value  of  each  input  variable 
used  in  the  construction  of  the  model.  The  input  variable  noise  is  not  included  in  Table 
4.1  because  we  appended  noise  to  each  unit  cost  at  different  levels. 


Table  4.1.  Fixed  Cost  Production  Simulation  Example  Input  Variables 


Variable 

Value 

Number  of  Lots  To  Be  Modeled 
(History) 

7 

Variable  Cost  Starting  Point 

$445,579 

Cumulative  Units 

301  units 

Learning  Slope 

84% 

Fixed  Cost  Burden 

20% 

Percent  Production  Decrease 

50% 

Production  Cost  Simulation:  Cost  Function  with  No  Fixed  Cost 
Our  research  focuses  on  modeling  a  production  decrease  and  the  inherent  loss  of 
efficiency  when  production  lines  must  adjust  to  unexpected  volume  variations.  The  plot 
of  our  second  cost  generating  function,  Figure  4.2,  does  not  include  a  fixed  cost  variable. 
Although  Figure  4.2’ s  plot  does  not  expose  a  loss  of  efficiency,  the  simulated  datasets 
maintain  the  validity  in  the  evaluation  of  the  cost  improvement  models.  When  we 
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compare  the  production  accuracy  from  this  production  model  to  the  fixed  cost  production 
results,  we  can  illustrate  the  behavior  of  each  model  under  varying  conditions. 


Figure  4.2:  Production  Simulation  Example  with  No  Fixed  Cost 


Historical  Production  Cost 

Our  fixed  cost  production  closely  aligns  with  the  hypothesized  patterns  described 
in  Chapter  III,  but  it  is  essential  that  we  demonstrate  with  certainty  that  our  production 
clearly  mimics  historical  patterns.  Utilizing  the  normalized  dataset  provided  by  the  Air 
Force  Cost  Analysis  Agency  (AFCAA),  we  can  successfully  model  a  program’s 
production  decrease.  The  F-15  program  suffered  production  decreases  in  1981  and  1982; 
Figure  4.3  shows  a  plot  of  the  lot  midpoints  and  average  unit  costs  for  the  program  from 
1973  through  these  production  decreases.  The  pattern  in  Figure  4.3  similarly  conforms  to 
production  costs  from  our  simulation  where  the  production  decrease  induces  the  average 
unit  cost  for  those  respective  lots  to  correspondingly  increase.  We  cannot  be  certain  of 
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the  true  extent  to  which  a  fixed  cost  burden  existed  in  the  F-15  program  at  that  time,  nor 
ascertain  the  exact  cause  of  the  rise  in  average  unit  cost,  but  Figure  4.3  does  reveal  a 
significant  loss  of  efficiency.  By  ensuring  that  our  production  datasets  mimic  patterns 
found  in  historical  programs,  we  feel  confident  that  the  findings  can  be  generalized 
beyond  simulated  data. 

The  lot  quantities  associated  with  the  F-15  program  fall  outside  of  the  range  of 
our  simulated  conditions,  but  by  matching  the  lot  quantities  we  can  create  cost  data  that 
almost  exactly  matches  the  F-15  scenario.  While  Figure  4.3  reveals  actual  historical  data, 
we  have  also  added  a  simulated  cost  profile  to  demonstrate  the  validity  of  our  Monte 
Carlo  simulation. 


0  100  200  300  400  500  600  700  800 

Unit  Number 

F-15  Actual  Cost  •  •  •  •  Simulation  Cost  Data 

Figure  4.3:  F-15  (1973-1982)  Flyaway  Production  Cost  (BY03  Million  $’s)  Compared  to  Simulated 

Cost  Data 

Though  the  profiles  do  not  exactly  mimic  each  other,  they  demonstrate  similar 
cost  patterns  and  the  cost  data  is  represented  by  a  4%  mean  absolute  percentage 
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difference.  Table  4.2  below  shows  the  simulated  conditions  that  created  the  cost  profile 


portrayed  above.  Each  of  the  independent  variables  falls  within  the  range  used  for  our 
simulations  shown  in  Chapter  III,  thus  our  simulated  data  accurately  represents  historic 
cost  data. 


Table  4.2.  Simulated  conditions  Creating  Historically  Accurate  Cost  Profile 


Simulated  Variable 

Variable  Level 

Number  of  Lots 

10 

Unit  1  Cost  (Million  $'s) 

$39 

Variable  Learning  Rate 

94% 

Fixed  Cost  Burden 

15% 

Production  Decrease 

44% 

Independent  Variable  Simulation  Results 

When  implementing  Monte  Carlo  simulation,  each  of  the  independent  variables 
produces  expected  values.  One  of  our  simulations  for  learning  curve  slope  represents  a 
triangular  distribution  with  a  minimum  value  of  75  percent,  a  mode  of  85  percent  and  a 
maximum  value  of  95  percent.  The  resulting  simulation  of  1,000  iterations  produces  the 
following  results. 
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Figure  4.4:  Learning  Curve  Distribution  from  Simulation  (1,000  Iterations) 

Distributions  for  the  remaining  generated  independent  variables  are  displayed  in 

Appendix  B. 

Model  Comparison 

Production  with  a  Fixed  Cost  Variable 

We  model  the  fixed  cost  production  simulation  example  shown  above  (Figure  4.1) 
with  the  results  from  the  Crawford  unit  cost  model,  production  rate  model,  production 
rate  adjustment  model,  and  the  fixed  cost  model.  Table  4.3  and  Table  4.4  present  the 
simulated  production  costs  and  the  model  results.  Though  Table  4.3  and  Table  4.4  are 
only  one  example  of  the  many  different  production  runs  we  completed  and  modeled,  this 
particular  data  articulates  patterns  consistent  throughout  our  simulations  and  modeling. 
Table  4.3  displays  the  dollar  values  from  each  of  the  models  and  Table  4.4  presents  the 
percentage  error  so  that  the  differences  among  the  models  can  be  clearly  established. 
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Table  4.3.  Model  Comparison  Example  (Constant  Year  $’s) 


Lot 

Lot 

Midpoint 

Actual  Avg. 
Unit  Cost 

Crawford  Unit 

Cost  Model 

Production 

Rate  Model 

Production 

Rate 

Adjustment 

Model 

Fixed  Cost 

Model 

1 

11 

$346,746 

$345,194 

$344,792 

$344,910 

$348,211 

2 

49 

$241,971 

$243,422 

$247,087 

$247,206 

$235,854 

3 

86 

$219,049 

$213,162 

$217,931 

$218,102 

$220,927 

4 

125 

$200,380 

$195,175 

$199,462 

$199,417 

$196,864 

5 

167 

$187,576 

$182,304 

$186,615 

$186,512 

$186,173 

6 

211 

$176,854 

$172,642 

$177,118 

$177,012 

$180,962 

7 

256 

$172,069 

$165,033 

$169,588 

$169,472 

$176,315 

8  (Estimated 
Lot) 

289 

$227,575 

$160,271 

$172,406 

$232,081 

$357,259 

Table  4.4.  Model  Comparison  Example:  Fit  and  Forecast  %  Error 


Lot 

Lot  Midpoint 

Crawford  Unit 

Cost  Model 

Production  Rate 

Model 

Production  Rate 

Adjustment 

Model 

Fixed  Cost 

Model 

1 

11 

0.45% 

0.56% 

0.53% 

-0.42% 

2 

49 

-0.60% 

-2.11% 

-2.16% 

2.53% 

3 

86 

2.69% 

0.51% 

0.43% 

-0.86% 

4 

125 

2.60% 

0.46% 

0.48% 

1.75% 

5 

167 

2.81% 

0.51% 

0.57% 

0.75% 

6 

211 

2.38% 

-0.15% 

-0.09% 

-2.32% 

7 

256 

4.09% 

1.44% 

1.51% 

-2.47% 

8  (Estimated 
Lot) 

289 

29.57% 

24.24% 

-1.98% 

-56.99% 

In  this  example  the  production  rate  adjustment  model  predicts  future  cost  more 
accurately  than  does  any  of  the  other  models.  Throughout  our  analysis  each  equation 
proves  to  possess  forecasting  capabilities;  each  equation  predicts  better  than  other  models 
on  specific  generated  iterations,  but  some  models  prove  to  produce  more  consistent 
forecasts. 
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Model  Fit  Accuracy 

Fit  accuracy  does  not  appear  to  significantly  vary  between  the  models.  Table  4.4 
establishes  that  each  of  the  models  fits  the  data  for  lots  1  through  7  extremely  well,  and 
that  each  of  the  simulations’  iterations  under  the  different  assumptions  evidences  the 
same  results.  Plots  depicting  the  cumulative  percentages  of  model  accuracy  for  the 
production  rate  adjustment  equation  and  fixed  cost  equation  for  the  first  set  of  simulation 
assumptions  (triangular  distribution  learning  slope  75.0  percent  to  95.0  percent  and  noise 
continuous,  uniform  distribution  -5.0  percent  to  15.0  percent)  verify  that  each  of  the 
equations  fits  the  data  well.  Plots  for  the  Crawford  unit  cost  model  and  the  production 
rate  model  can  be  viewed  in  Appendix  C.  The  Crawford  unit  cost  model  and  the 
production  rate  model  produce  similar  results.  Figure  4.5  and  other  plots  of  cumulative 
percentages  illustrate  that  at  any  chosen  point  along  the  y-axis,  we  can  be  a  certain 
percentage  confident  (y-axis  value)  that  the  model  error  will  be  less  than  or  equal  to  the 
value  on  the  x-axis.  When  comparing  the  models,  the  model  with  a  curve  appearing 
further  to  the  left  at  the  higher  cumulative  percentage  adduces  a  lower  error  from  actual 
values.  Thus,  in  Figure  4.5  the  models  fit  the  data  relatively  closely,  but  the  fixed  cost 
model  offers  us  a  higher  confidence  that  the  model  will  provide  a  lower  fit  error. 
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Fit  Absolute  Mean  %  Error 


Fixed  Cost  Model  Fit  Absolute  Mean  %  Error  Cumulative  Percentages 

Production  Rate  Adjustment  Model  Fit  Absolute  Mean  %  Error 
Cumulative  Percentages 


Figure  4.5:  Fixed  Cost  and  Production  Rate  Adjustment  Model  Fit  Absolute  Mean  %  Errors  for 

Fixed  Cost  Production 

The  cumulative  percentages  for  fit  error  confirm  that  each  of  the  models  fit  the 
data  extremely  well.  Statistically,  a  test  of  equal  means  for  the  absolute  mean  percentage 
fit  error  establishes  that  the  means  are  not  equal,  but  the  cumulative  chart  verifies  that  the 
models  all  fit  the  data  well.  Table  4.4  discloses  the  cumulative  percentages  of  the  four 
models.  When  translated,  the  percentages  mean  that  we  can  be  confident  by  a  certain 
defined  percentage  that  the  model  will  return  a  fit  error  equal  to  or  less  than  that 
identified  fit  error.  For  example,  for  the  Crawford  unit  cost  model,  we  can  be  75  percent 
confident  that  the  fit  error  for  any  model  will  be  less  than  or  equal  to  2.0  percent  under 
these  simulated  conditions. 
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Table  4.5.  Fit  Absolute  Mean  %  Error  Cumulative  Percentages  for  Production  with  Fixed  Costs 


Cumulative 

Percentage 

Crawford  Unit 

Cost  Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

25% 

1% 

1% 

1% 

1% 

50% 

1% 

1% 

1% 

2% 

75% 

2% 

2% 

2% 

4% 

95% 

26% 

14% 

14% 

12% 

The  production  rate  model  and  the  production  rate  adjustment  model  have  the 
same  fit  errors  because  they  fit  data  applying  the  same  model  parameters.  The  slight 
deviation  shown  in  Table  4.4  stems  from  very  small  differences  in  the  model  parameters 
ascertained  by  Solver  (rounding).  Running  each  of  the  equations  separately  with  Solver 
to  ensure  the  program  identifies  the  same  parameters,  the  equations  perform  properly. 
We  find  that  even  though  there  are  slight  differences,  the  means  of  each  of  the  fit  errors 
are  statistically  equal.  The  differences  between  the  two  models  can  be  verified  during 
estimation  where  the  production  rate  adjustment  model  adds  the  variable  to  account  for 
changes  in  lot  quantities. 

Under  different  assumptions,  the  fit  errors  yield  similar  results.  We  do  not 
uncover  any  significant  differences  in  fit  error  when  the  independent  variables  vary;  the 
fit  errors  remain  consistently  low.  Figure  4.8  evinces  the  fixed  cost  model  fit  absolute 
mean  percentage  errors  for  our  production  simulation  without  a  fixed  cost  variable; 
Figure  4.9  illustrates  the  fit  error  for  the  production  rate  adjustment  model.  This 
production  simulation  differs  only  by  the  exclusion  of  the  fixed  cost  variable;  all  other 
independent  variables  are  simulated  with  the  same  ranges. 
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Fit  Absolute  Mean  %  Error 


Fixed  Cost  Model  Fit  Absolute  Mean  %  Error  Cumulative  Percentages 

Production  Rate  Adjustment  Model  Fit  Absolute  Mean  %  Error 
Cumulative  Percentages 


Figure  4.6:  Fixed  Cost  and  Production  Rate  Adjustment  Model  Fit  Absolute  Mean  %  Errors  For 

Production  without  Fixed  Costs 

The  plots  appear  almost  identical  to  the  fit  error  for  the  production  simulation 
with  a  fixed  cost  variable  because  the  major  cost  changes  do  not  surface  until  the 
production  decrease  occurs  in  the  estimated  lot.  The  cumulative  percentages  for  the  fit 
absolute  mean  percentage  errors  for  each  of  the  four  models  are  the  focus  subject  of 
Table  4.6. 

Table  4.6.  Fit  Absolute  Mean  %  Error  Cumulative  Percentages  for  Production  without  Fixed  Costs 


Cumulative 

Percentage 

Crawford  Unit  Cost 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment 

Model 

Fixed  Cost 

Model 

25% 

1% 

1% 

1% 

2% 

50% 

1% 

1% 

1% 

4% 

75% 

1% 

2% 

2% 

7% 

95% 

57% 

16% 

16% 

17% 
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The  fixed  cost  model  yields  a  slightly  higher  fit  error  than  do  the  other  three  models  until 


the  95%  cumulative  percentage  where  it  demonstrates  a  significantly  better  fit  than  do  the 
other  models.  Crawford’s  unit  cost  model  strictly  mimics  the  cost  production  function 
we  simulate  except  for  the  error  term.  As  a  result,  we  expect  a  low  fit  error. 

The  fit  errors  for  each  of  the  models  prove  to  be  very  low,  and  each  of  the  models 
performs  equally  well.  The  major  differences  among  the  models  are  evidenced  during 
forecasting  where  we  discover  truly  significant  differences  among  the  models. 

Model  Forecasting  Error 

We  measure  the  mean  percentage  error  for  each  of  the  model  forecasts  to 
determine  if  any  of  the  models  demonstrates  a  consistent  bias  to  over  or  underestimate 
future  production  cost.  Table  4.7  catalogs  the  results  for  each  of  the  models  from  the 
production  simulation  with  a  fixed  cost  component. 


Table  4.7.  Model  Mean  Percentage  Error  for  the  Production  with  Fixed  Cost 


Crawford  Unit  Cost 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost  Model 

Forecast  MPE 

36% 

19% 

-209% 

-45% 

Table  4.7  provides  a  snapshot  of  each  of  the  models  to  determine  whether  or  not  the 
individual  model  tends  to  over  or  underestimate  production  cost.  The  value  shown  in  the 
table  is  an  average,  and  we  discover  that  the  value  can  be  influenced  by  extremely  large 
values.  Obviously,  as  a  number  based  on  1,000  observations,  an  outlier  will  prove  to 
exhibit  less  influence.  Table  4.7  reveals  that  the  Crawford  unit  cost  model  and  the 
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production  rate  model  tend  to  underestimate,  while  the  production  rate  adjustment  model 


and  the  fixed  cost  model  tend  to  overestimate. 

Our  next  step  is  to  measure  the  forecast  mean  absolute  percentage  error.  Just  as 
with  the  model  fit  errors,  we  look  at  the  cumulative  percentages  to  determine  our 
confidence  levels  in  individual  models.  Table  4.8  presents  the  absolute  percentage  errors 
for  all  four  models  when  forecasting  the  production  cycle  including  a  fixed  cost 
component.  The  corresponding  cumulative  percentage  histogram  plots  can  be  seen  in 
Appendix  D. 

Table  4.8.  Forecast  Absolute  Mean  %  Error  Cumulative  Percentages  for  Production  With  Fixed 

Costs 


Cumulative 

Percentage 

Crawford  Unit 

Cost  Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

25% 

18% 

9% 

20% 

17% 

50% 

33% 

18% 

54% 

33% 

75% 

55% 

36% 

161% 

65% 

95% 

69% 

57% 

861% 

154% 

Each  model  presents  data  points  that  appear  to  be  outliers.  For  example,  with  the 
production  rate  model,  we  find  a  model  with  a  fit  absolute  mean  percentage  error  of  28 
percent  and  a  corresponding  forecast  absolute  mean  percentage  error  of  152  percent. 

This  data  point  proves  to  be  the  largest  for  the  production  rate  model  by  approximately 
50  percent.  To  test  the  effects  on  the  cumulative  percentages,  we  remove  this  data  point 
and  recalculate  the  percentages.  The  resulting  cumulative  percentages  do  not  change 
from  9  percent,  18  percent,  and  36  percent.  So,  even  though  each  of  the  models  produces 
some  estimates  that  appear  to  be  outliers,  those  values  provide  minimal,  if  any,  impact  on 
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our  overall  evaluations  of  the  models.  We  do  not  remove  any  apparent  outliers  because 
the  points  do  not  affect  the  overall  results,  and  Solver  finds  the  apparent  outlier  models  to 
best  fit  the  generated  data. 

Based  on  the  forecast  errors  for  the  production  simulation  with  a  fixed  cost 
component,  the  production  rate  model  outperforms  the  other  three  models.  Using  our 
second  cost  production  function,  we  discover  similar  results.  Table  4.9  provides  the 
mean  percentage  error  for  each  of  the  models  to  measure  any  bias  across  all  estimates. 
Table  4.10  shows  the  forecast  error  cumulative  percentages  for  each  of  the  models  when 
the  production  simulation  does  not  include  a  fixed  cost  variable.  Because  the  production 
rate  model  so  closely  mirrors  the  Crawford  unit  cost  model,  except  for  the  Noise  variable, 
we  expect  the  Crawford  model  to  outperform  the  other  models.  We  use  the  Crawford 
model  as  a  check  on  our  simulation  and  as  a  comparison  tool  for  our  other  models.  Table 
4.8  shows  that  the  Crawford  model  slightly  underestimates  cost  while  each  of  the  other 
models  drastically  overestimates  cost. 


Table  4.9.  Model  Mean  Percentage  Error  For  Production  without  Fixed  Cost 


Crawford  Unit  Cost 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

Estimate  MPE 

7% 

-20% 

-292% 

-152% 
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Table  4.10.  Forecast  Mean  Absolute  %  Error  Cumulative  Percentages  for  Production 


without  Fixed  Costs 


Cumulative 

Percentage 

Crawford  Unit  Cost 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

25% 

1% 

5% 

34% 

49% 

50% 

1% 

13% 

93% 

109% 

75% 

2% 

29% 

212% 

260% 

95% 

74% 

60% 

814% 

392% 

As  expected,  the  Crawford  unit  cost  model  produces  accurate  cost  estimates.  The 
production  rate  model  produces  very  consistent  results  demonstrating  similar  accuracy 
with  both  cost  production  functions.  The  production  rate  adjustment  model  and  fixed 
cost  model  do  not  predict  cost  accurately  with  this  function.  Each  of  these  models 
hypothetically  accounts  for  changes  in  the  normal  production  pattern,  and  this  cost 
function  does  not  create  changes.  These  two  models  should  not  be  used  if  production 
remains  consistent  because  they  will  model  aspects  of  production  that  are  not  present  and 
will  thus  create  inaccurate  forecasts. 

Model  Performance  under  Differing  Simulated  Conditions 
Monte  Carlo  simulation  allows  us  to  vary  the  assumptions  of  our  production  cost 
profiles  to  evaluate  each  of  the  model’s  reaction,  if  any,  to  varying  conditions.  We  vary 
the  learning  curve  slope  as  well  as  the  noise  distribution  to  simulate  different  production 
costs,  neither  of  which  produces  results  different  from  those  previously  presented. 

Overall,  the  production  rate  model  outperforms  the  other  models.  This  does  not 
hold  true  when  we  isolate  the  variables  and  the  fixed  cost  burden,  nor  when  production 
decreases.  We  notice  that  even  though  the  production  rate  model  appears  to  forecast 
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better  then  each  of  the  other  models,  production  with  a  high  fixed  cost  variable  causes  the 
fixed  cost  model  to  forecast  equally  well  and  even  considerably  better  under  some 
circumstances.  As  a  reminder,  our  fixed  cost  variable  captures  the  inefficiency  that  can 
be  gained  when  production  faces  an  unplanned  decrease  in  the  number  of  units  produced 
in  a  lot.  The  inefficiencies  can  include  fewer  units  to  absorb  high  fixed  costs,  penalties 
from  suppliers  for  reducing  order  quantities,  overhead  associated  with  a  production  plant, 
costs  associated  with  re-tooling  the  production  line,  labor  costs  associated  with  re-tooling 
the  production  line,  and  loss  of  learning  because  the  labor  force  must  adjust  to  the  new 
conditions.  We  cannot  simulate  each  of  these  possible  scenarios  individually,  so  we 
capture  the  related  costs  in  our  fixed  cost  variable.  This  variable  affects  both  the 
production  costs  and  the  model’s  performance. 

Table  4.11  displays  the  forecast  absolute  mean  percentage  error  cumulative 
percentages  for  production  with  a  fixed  cost  burden  of  40  percent,  the  highest  fixed  cost 
burden  we  simulate. 

Table  4.11.  Forecast  Absolute  Mean  %  Error  Cumulative  Percentages  for  Production  with  a  40% 


Fixed  Cost  Burden 


Cumulative 

Percentage 

Crawford  Unit 

Cost  Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

25% 

21% 

11% 

24% 

9% 

50% 

37% 

23% 

59% 

20% 

75% 

62% 

45% 

169% 

35% 

95% 

69% 

60% 

832% 

58% 

The  data  display  in  Table  4.1 1  substantiates  that  at  25  percent,  50  percent,  and  75 
percent  cumulative  percentages,  we  can  be  confident  that  the  fixed  cost  model  will 
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generate  a  more  accurate  forecast  than  does  any  other  model.  In  fact,  further  simulation 


reveals  that  a  fixed  cost  burden  of  33  percent  is  the  infliction  point  between  the  fixed  cost 
model  and  the  production  rate  model.  The  production  rate  model  will  produce  more 
accurate  estimates  when  the  fixed  cost  burden  falls  below  33  percent,  and  the  fixed  cost 
model  creates  more  accurate  estimates  when  fixed  costs  exceed  33  percent.  Analysis 
shows  that  the  MAPE  of  the  fixed  model  is  negatively  correlated  with  fixed  cost  burden 
at  a  level  of  -.5.  This  value  means  that  as  fixed  cost  rises,  the  MAPE  will  become  smaller 
(the  model  becomes  more  accurate).  The  Crawford  unit  cost  model  and  the  production 
rate  model  show  a  slight  positive  correlation  to  fixed  cost,  .3  and  .2  respectively;  each  of 
these  models  demonstrates  less  accuracy  as  fixed  costs  rise.  The  production  rate 
adjustment  model  did  not  show  any  correlation  to  the  fixed  cost  variable. 

When  we  look  even  further  into  the  simulated  conditions,  we  notice  that  the 
amount  of  production  decrease  also  affects  which  model  establishes  the  most  accurate 
estimations. 

Table  4.12.  Forecast  Absolute  Mean  %  Error  Cumulative  Percentages  for  Production  with  a  40% 
Fixed  Cost  Burden  and  A  50%  Production  Decrease 


Model  Error 

Cumulative 

Percentage 

Crawford  Unit  Cost 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost  Model 

25% 

35% 

14% 

12% 

50% 

38% 

24% 

68% 

25% 

75% 

41% 

33% 

178% 

34% 

95% 

46% 

43% 

845% 

42% 
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Modeling  Historical  Data 

We  model  four  sets  of  data  provided  by  the  AFCAA.  As  previously  discussed, 
the  data  has  been  normalized  by  RAND  to  return  the  dollar  figures  to  base  year  and  to 
remove  any  anomalies  that  should  not  be  modeled  as  pattern.  In  each  of  these  four  sets  of 
data,  the  respective  program  suffers  a  production  decrease  resulting  in  an  increase  to  the 
average  unit  cost  for  the  next  production  lot.  The  programs  where  we  observed  this 
pattern  are  the  F-15  program,  the  F-16  program,  and  the  F-18  program.  None  of  the 
production  decreases  come  at  the  end  of  the  production  life  cycle;  the  decreases  occur 
within  the  first  10  years  of  the  program.  We  model  the  F-15  twice  having  observed  that 
the  program  suffers  a  minor  production  decrease,  which  is  then  followed  the  next  year  by 
a  significantly  larger  decrease  in  production.  Using  the  small  decrease  in  production 
during  the  model  fit  provides  us  with  information  about  how  the  models  react  to  the 
changes  and  then  forecast  the  changes  to  follow  in  the  next  year.  Each  of  the  historical 
datasets  is  displayed  in  Appendix  G. 

Modeling  historical  data  verifies  our  findings  from  the  simulated  data.  Most 
importantly,  this  modeling  certifies  that  the  fixed  cost  model  outperforms  any  of  the  other 
models  overall.  We  anticipate  this  result;  when  these  military  aircraft  production 
programs  suffer  sudden  yield  decreases,  there  are  major  losses  of  efficiency  that  cause 
average  unit  costs  to  increase.  The  production  rate  model  also  performs  well  with 
estimation  results  quite  similar  to  the  fixed  cost  model.  Table  4.13  exhibits  the 
percentage  error  for  each  of  the  models  and  reveals  that  the  production  rate  adjustment 
model  is  the  only  model  manifesting  a  significant  bias.  The  resultant  negative  values 
mean  that  the  model  tends  to  overestimate  consistently  across  each  of  the  datasets.  No 
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other  model  produces  an  overwhelming  bias  in  any  direction  as  can  be  seen  by  the 
averages  at  the  bottom  of  each  column. 

Table  4.13.  Model  Percentage  Error  for  Each  Historical  Dataset 


Crawford 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

F-15 

-1% 

11% 

-36% 

11% 

25% 

9% 

19% 

-1% 

F-16 

15% 

9% 

-80% 

-11% 

F-18 

-16% 

-21% 

-113% 

-17% 

Average 

6% 

2% 

-53% 

-4% 

When  looking  at  the  absolute  percentage  error  for  each  of  the  models,  the  fixed 
cost  model  does  not  outperform  the  other  models  in  every  instance,  but  it  does 
outperform  the  other  models  overall.  While  the  Crawford  unit  cost  model  and  the 
production  rate  model  do  provide  lower  errors  at  times,  the  fixed  cost  model  establishes  a 
lower  average  across  all  four  datasets.  The  averages  shown  at  the  bottom  of  Table  4.14 
indicate  how  each  of  the  models  estimates  the  historical  data. 


Table  4.14.  Model  Absolute  Percentage  Error  for  Each  Historical  Dataset 


Crawford 

Model 

Production  Rate 

Model 

Production  Rate 
Adjustment  Model 

Fixed  Cost 

Model 

F-15 

1% 

11% 

36% 

11% 

25% 

9% 

19% 

1% 

F-16 

15% 

9% 

80% 

11% 

F-18 

16% 

21% 

113% 

17% 

Average 

14% 

12% 

62% 

10% 

Although  these  results  are  only  snapshots  of  a  couple  of  programs,  the  historical 
patterns  and  the  model  results  match  the  findings  from  our  simulated  data.  When  a 
program  suffers  a  production  decrease  that  causes  average  unit  cost  to  increase,  the  fixed 
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cost  model  provides  more  accurate  estimates  than  do  any  of  the  other  four  models  in  our 
research.  We  are  aware  that  every  program  contains  specific  attributes  and  cost  drivers 
that  might  only  be  found  through  a  grassroots  cost  analysis,  but  our  research  conceivably 
can  provide  some  insight  into  the  behaviors  of  cost  with  a  production  decrease. 

Chapter  Summary 

Our  Monte  Carlo  simulation  for  production  costs  closely  resembles  patterns  found 
within  historical  data.  While  each  of  the  cost  improvement  models  fits  the  data 
extremely  well,  the  forecast  errors  greatly  differ  among  the  models.  Throughout  all  of 
the  simulations’  conditions,  the  production  rate  model  forecasts  with  the  least  error,  and 
thus  outperforms  other  cost  improvement  models.  When  programs  suffer  from  high  fixed 
costs,  the  fixed  cost  model  captures  the  inefficiencies  with  production  decreases  and 
forecasts  equally  as  well  as  the  production  rate  model.  Though  the  fixed  cost  model 
appears  to  forecast  slightly  better  for  a  program  with  high  losses  of  efficiency, 
statistically  the  absolute  mean  percentage  forecast  errors  are  equal  for  both  models.  The 
historical  datasets  validate  our  findings  and  actually  reveal  that  the  fixed  cost  model 
outperforms  each  of  the  other  models.  In  Chapter  V  we  discuss  the  strengths,  limitations, 
policy  implications,  and  possible  future  research  based  upon  our  research. 
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V.  Conclusions 


Chapter  V  first  focuses  on  the  strengths  of  our  research;  we  cover  the  highlights 
from  Chapter  IV  and  the  most  essential  takeaways.  Second,  we  look  at  the  limitations  of 
our  research.  Though  we  identified  our  limitations  in  Chapter  I,  we  feel  that  we  need  to 
revisit  these  limitations  due  to  their  significance  to  the  results  of  our  research.  Third,  we 
discuss  the  policy  implications  of  our  research  followed  by  possible  future  research  based 
upon  our  findings. 

Strengths 

Our  main  focus  in  our  research  is  to  evaluate  cost  improvement  models  and 
challenge  the  status  quo  of  models  used  in  estimating.  In  an  environment  of  restricted 
budgets,  short  deadlines,  and  production  decreases,  coupled  with  increased  pressure  to 
produce  accurate  estimates,  the  introduction  of  a  fixed  cost  model  offers  opportunities  for 
more  accurate  estimates  under  certain  program  conditions.  When  a  program  suffers  a 
production  decrease  and  a  subsequent  lost  of  efficiency,  the  fixed  cost  model  provides 
more  accurate  estimates  than  do  other  models.  The  loss  of  efficiency  can  be  due  to  high 
fixed  costs  or  changes  in  the  production  line,  and  that  impediment  results  in  higher 
average  unit  cost  for  the  reduced  production  lot.  Based  on  our  analysis  of  historical  data, 
loss  of  efficiency  took  place  within  past  programs  and  estimates  could  have  benefited 
from  utilization  of  the  fixed  cost  model.  We  also  clearly  illustrate  that  over  the  entire 
range  of  possible  programs  and  cost  profiles,  the  production  rate  model  provides  the  most 
accurate  estimates.  We  reach  these  conclusions  through  thousands  of  iterations  of 
possible  cost  profiles  and  by  altering  variables  affecting  cost  for  these  iterations. 
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We  built  our  research  on  a  solid  foundation  of  academics  that  scrutinized  similar 


topics.  Our  literature  review  discusses  in  depth  the  works  of  other  authors  who  explored 
cost  improvement  curves,  and  we  use  that  research  as  a  foundation  for  our  methodology. 
The  confidence  in  our  results  also  stems  from  the  thoroughness  of  our  Monte  Carlo 
simulation  and  our  ability  to  create  a  wide  range  of  possible  production  cost  patterns. 
Varying  independent  variables  through  thousands  of  possible  combinations  creates  an 
accurate  depiction  of  how  each  of  the  models  estimates.  Isolating  changes  in  the 
independent  variables  allows  us  to  detect  situations  where  the  fixed  cost  model  furnishes 
more  accurate  estimates  than  do  the  other  models.  We  hypothesized  that  during  a 
production  decrease  with  a  large  loss  of  efficiency,  the  fixed  cost  model  would 
outperform  other  models;  our  analysis  proves  this  to  be  true.  By  offering  more  insight 
into  cost  improvement  modeling,  we  hope  that  cost  analysts  are  able  to  dispense  more 
timely  and  accurate  forecasts  to  aid  the  defense  acquisition  system  in  delivering  assets  to 
the  warfighter. 

Limitations 

Though  we  are  confident  that  our  Monte  Carlo  simulation  covers  a  wide  range  of 
potential  cost  profiles,  we  cannot  possibly  simulate  every  practicable  condition.  We  feel 
that  we  have  generalized  the  simulation  to  focus  on  the  more  common  cost  drivers,  but 
other  exogenous  factors  can  engender  changes  in  cost  and  our  simulation  does  not  deal 
with  these  inconsistencies.  We  assume  that  our  dataset  has  been  normalized  so  that  all 
inconsistencies  have  been  eliminated,  but  in  reality  each  program  exhibits  unforeseen 
uniqueness. 
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We  are  also  limited  by  the  actuality  that  we  cannot  evaluate  every  potential  cost 
improvement  model  available  to  cost  analysts.  There  could  be  circumstances  where 
another  model  outperforms  our  models,  but  we  chose  to  restrict  our  research  due  to  the 
infeasibility  of  evaluating  every  model  available. 

Policy  Implications 

We  hope  to  enhance  materials  such  as  The  Air  Force  Cost  Analysis  Handbook 
such  that  they  furnish  more  specific  direction  to  cost  estimators.  The  flexibility  within 
resources  forces  analysts  to  explore  available  methods  and  discover  the  most  appropriate 
models  for  their  respective  programs,  but  sometimes  the  expansive  options  impede 
discovery.  Searching  all  possible  avenues  for  cost  estimation  takes  time  not  often 
realistic  in  these  pressurized,  time-conscious  situations.  Our  research  acts  as  a  more 
accurate  ‘guide’  for  costs  analysts.  While  other  cost  improvement  models  should  not  be 
discounted  early  in  the  process,  our  research  identifies  the  validity  of  the  fixed  cost  model 
and  the  need  for  the  model’s  inclusion  in  common  practice.  Such  is  the  case  especially 
when  programs  face  reductions  as  the  F-22  program  currently  faces.  Instructional 
materials  presenting  the  Crawford  unit  cost  model,  the  production  rate  model,  and  the 
production  rate  adjustment  model  need  to  also  include  the  fixed  cost  model.  We  have 
constructed  a  foundation  elucidating  circumstances  where  the  fixed  cost  model  proves 
most  useful;  dissemination  of  that  information  is  essential. 

Further  Research 

More  intensive  examination  should  be  done  regarding  application  of  the  fixed 
cost  model.  First,  the  model  needs  to  be  explored  with  specific  datasets  to  check  the 
statistical  characteristics  of  the  model.  Our  research  did  not  include  statistical  validation 
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of  the  model  due  to  the  voluminous  number  of  different  datasets  we  modeled.  In  order  to 


further  the  validation  of  the  model,  statistical  characteristics  should  be  studied.  Our 
research  explores  ‘at  point’  estimates  with  the  fixed  cost  model,  but  a  more  in-depth 
analysis  of  the  model  should  target  the  behavior  of  confidence  intervals. 

Further  Research  could  direct  efforts  toward  discovering  a  more  accurate  cost 
improvement  model.  As  conditions  change  and  programs  face  new  challenges,  the 
current  cost  improvement  models  may  not  be  sufficient.  None  of  the  models  we  evaluate 
present  an  overwhelming  accuracy  with  either  the  simulated  or  the  historical  data.  Thus, 
perhaps  a  newly  formulated  model  would  replicate  and  project  cost  more  accurately.  Our 
research  and  further  research  should  re-examine  cost  analysis  methodologies  and  strive  to 
improve  current  methods  of  cost  estimating  and  to  provide  more  complete  and  accurate 
estimates. 
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Appendix  A:  Fixed  Cost  Production  Function  Simulation  Example  Dataset 


Table  A.l.  Fixed  Cost  Production  Function  Simulation  Example 


Lot  Number 

Lot 

Quantity 

Cum 

Quantity 

Lot 

Midpoint 

Actual  Lot  Cost 

($) 

Lot  Average  Unit  Cost 

(S) 

1 

32 

32 

11.08 

11,095,882 

346,746 

2 

36 

68 

48.94 

8,710,967 

241,971 

3 

37 

105 

86.06 

8,104,799 

219,049 

4 

41 

146 

125.20 

8,215,572 

200,380 

5 

43 

189 

167.34 

8,065,774 

187,576 

6 

44 

233 

210.95 

7,781,591 

176,854 

7 

45 

278 

255.53 

7,743,111 

172,069 

8 

(Estimated 

Lot) 

22 

300 

289.40 

5,006,658 

227,575 
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Appendix  B:  Independent  Variable  Distributions  from  Monte  Carlo  Simulation 


10%  20%  30%  40% 

Fixed  Cost  Burden 

■  Fixed  Cost  Burden  Percentage  Frequency 


Figure  B.l.  Fixed  Cost  Burden  Distribution  From  Simulation  (1,000  Iterations) 


■  Percentage  Production  Decrease  Frequency 


Figure  B.2:  Production  Decrease  Distribution  From  Simulation  (1,000  iterations) 
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Frequency 


Unit  One  Cost  (Constant  Year  $'s) 

■  Unit  One  Cost  Frequency 


Figure  B.4:  Unit  One  Cost  Distribution  from  Simulation  (1,000  Iterations) 
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Figure  B.6:  Cumulative  Quantities  From  Simulation  (1,000  Iterations) 
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Frequency 


Appendix  C:  Model  Fit  Absolute  Mean  %  Error  Plots  From  Production  with  Fixed 

Cost  Variable 


Figure  C.l:  Production  Rate  Model  Fit  Absolute  Mean  %  Error  For  Production  With  Fixed 

Cost  Variable 


Figure  C.2:  Crawford  Unit  Cost  Model  Fit  Absolute  Mean  %  Error  For  Production  With 

Fixed  Cost  Variable 
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Fretluency  Frequency 


Appendix  D:  Model  Fit  Absolute  Mean  %  Error  Plots  From  Production  without 


Fixed  Cost  Variable 
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Figure  D.l:  Production  Rate  Model  Fit  Absolute  Mean  %  Error  From  Production  Without 


Fixed  Cost  Variable 
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Figure  D.2:  Crawford  Unit  Cost  Model  Fit  Absolute  Mean  %  Error  From  Production 
Without  Fixed  Cost  Variable 
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Appendix  E:  Model  Forecast  Absolute  Mean  %  Error  Plots  From  Production  with 

Fixed  Cost  Variable 


Figure  E.l:  Crawford  Unit  Cost  Model  Forecast  Absolute  Mean  %  Error  From  Production 

With  Fixed  Cost  Variable 


Figure  E.2:  Production  Rate  Model  Forecast  Absolute  Mean  %  Error  From  Production 

With  Fixed  Cost  Variable 


82 


Frequency  Frequency 


Figure  E.3:  Production  Rate  Adjustment  Model  Forecast  Absolute  Mean  %  Error  From 
Production  With  Fixed  Cost  Variable 


Figure  E.4:  Fixed  Cost  Model  Forecast  Absolute  Mean  %  Error  From  Production  With 

Fixed  Cost  Variable 
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Appendix  F:  Model  Forecast  Absolute  Mean  %  Error  Plots  From  Production 

without  Fixed  Cost  Variable 


Figure  F.l:  Crawford  Unit  Cost  Model  Forecast  Absolute  Mean  %  Error  From  Production 

Without  Fixed  Cost  Variable 


Figure  F.2:  Production  Rate  Model  Forecast  Absolute  Mean  %  Error  From  Production 
Without  Fixed  Cost  Variable 


84 


Frequency  Frequency 


Figure  F.3:  Production  Rate  Adjustment  Model  Forecast  Absolute  Mean  %  Error  From 
Production  Without  Fixed  Cost  Variable 


Figure  F.4:  Fixed  Cost  Model  Forecast  Absolute  Mean  %  Error  From  Production  Without 

Fixed  Cost  Variable 
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Appendix  G:  Historical  Datasets 


Table  G.l.  F-15  Dataset 


Year 

Lot  Quantity 

First  Unit 

Last  Unit 

Lot 

Midpoint 

Lot  Average 
Unit  Cost 
(BY03  Million 
$'s) 

1973 

30 

1 

30 

11.9 

40.0 

1974 

62 

31 

92 

58.7 

30.6 

1975 

72 

93 

164 

126.8 

26.8 

1976 

153 

165 

317 

236.8 

26.8 

1977 

108 

318 

425 

370.2 

25.5 

1978 

97 

426 

522 

473.2 

27.5 

1979 

91 

523 

613 

567.4 

25.1 

1980 

98 

614 

711 

662.0 

24.8 

1981 

Estimated 

Lot 

85 

712 

796 

753.7 

27.9 
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Table  G.2.  F-15  Dataset  (2) 


Year 

Lot  Quantity 

First  Unit 

Last  Unit 

Lot 

Midpoint 

Lot  Average 
Unit  Cost 
(BY03  Million 
$'s) 

1973 

30 

1 

30 

11.9 

40.0 

1974 

62 

31 

92 

58.7 

30.6 

1975 

72 

93 

164 

126.8 

26.8 

1976 

153 

165 

317 

236.8 

26.8 

1977 

108 

318 

425 

370.2 

25.5 

1978 

97 

426 

522 

473.2 

27.5 

1979 

91 

523 

613 

567.4 

25.1 

1980 

98 

614 

711 

662.0 

24.8 

1981 

85 

712 

796 

753.7 

27.9 

1982 

Estimated 

Lot 

38 

797 

834 

815.5 

32.4 

Table  G.3.  F-16  Dataset 


Year 

Lot 

Quantity 

First  Unit 

Last  Unit 

Lot  Midpoint 

Lot  Average 
Unit  Cost  (BY03 
Million  $'s) 

1978 

202 

1 

202 

76 

16.2 

1979 

250 

203 

452 

319 

12.3 

1980 

287 

453 

739 

590 

12.6 

1981 

276 

740 

1015 

874 

12.7 

1982 

242 

1016 

1257 

1134 

13.2 

1983 

168 

1258 

1425 

1341 

14.3 

87 


Table  G.4.  F-18  Dataset 


Year 

Lot 

Quantity 

First  Unit 

Last  Unit 

Lot  Midpoint 

Lot  Average 
Unit  Cost  (BY03 
Million  $'s) 

1979 

9 

1 

9 

4 

89.2 

1980 

25 

10 

34 

20 

60.9 

1981 

79 

35 

113 

69 

38.5 

1982 

87 

114 

200 

155 

33.5 

1983 

125 

201 

325 

260 

28.8 

1984 

134 

326 

459 

390 

26.2 

1985 

145 

460 

604 

530 

23.4 

1986 

138 

605 

742 

672 

22.6 

1987 

109 

743 

851 

796 

22.4 
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